Permalink
Browse files

Better example in the README file.

Changed to use Configurator.working_directory
Faster load time for Pipemaster client (no longer loads the world)
Documented Pipemaster::Client
Fixed: PIPE_ENV, not RACK_ENV
  • Loading branch information...
1 parent 746e590 commit 69ab6a8daa4224238dee5a2a7ba4a0f3171d2f8c @assaf committed Feb 18, 2010
Showing with 101 additions and 38 deletions.
  1. +37 −10 README.rdoc
  2. +5 −3 bin/pipemaster
  3. +5 −5 lib/pipemaster.rb
  4. +29 −5 lib/pipemaster/client.rb
  5. +16 −7 lib/pipemaster/configurator.rb
  6. +7 −7 lib/pipemaster/server.rb
  7. +1 −1 pipemaster.gemspec
  8. +1 −0 test/test_helper.rb
View
@@ -1,22 +1,49 @@
+You pipe from here, to a process that runs over there. Pipemaster forks,
+redirects and runs your command.
+
+== Why
+
+I've got a short task I want to perform. To validate an incoming email, parse
+the message and store the results in the database(*). I setup Postfix to pipe
+incoming emails into a Ruby script.
+
+Ruby processes are fairly cheap, until you get into loading the mail library,
+the database library, the ORM, the application logic, the ... you get the
+picture.
+
+I use Pipemaster to fire up the main process once, and the Pipemaster client to
+run commands on that server. Still have to make sure the code is light and
+fast, but I did eliminate the significant initialization overhead.
+
+As you can guess, Pipemaster supports piping input and output streams, and uses
+forking (sorry Windows; for JRuby see Nailgun).
+
+* Processing emails as they come allows the application to reject unauthorized
+ senders immediately by replying with an SMTP code. The alternative, accepting
+ the email and later on sending a bounce, leads to backscatter (see
+ http://en.wikipedia.org/wiki/Backscatter_(e-mail)).
+
+
== Using Pipemaster
-1. Create a Pipemaster file. For example:
+Step 1: Create a Pipemaster file. For example:
#!highlight/ruby
- require File.dirname(__FILE__) + '/config/environment'
-
- command :test do |*args|
- puts [$stdin.read.strip, Rails.version].join(" ")
+ command :echo do |*args|
+ first, rest = $stdin.read.split
+ $stdout << [first, args, rest].flatten.join(" ")
end
-2. Start the Pipemaster server:
+Step 2: Fire up the Pipemaster server:
$ pipemaster --server
+ I, [2010-02-18T12:57:57.739230 #5460] INFO -- : master process ready
+ I, [2010-02-18T12:57:57.739606 #5460] INFO -- : listening on addr=127.0.0.1:7887 fd=3
+
+Step 3: For a new shell, execute a command:
-3. Execute a command:
-
- $ echo "This is Rails " | pipemaster test
- This is Rails 2.3.5
+ $ echo "Stand down!" | ruby -Ilib bin/pipemaster echo upside
+ Stand upside down!
== License
View
@@ -3,12 +3,11 @@
require "pipemaster"
require "optparse"
-ENV["RACK_ENV"] ||= "development"
+ENV["PIPE_ENV"] ||= "development"
server = false
daemonize = false
listeners = []
options = { :listeners => listeners }
-host, port = Pipemaster::DEFAULT_HOST, Pipemaster::DEFAULT_PORT
opts = OptionParser.new("", 24, ' ') do |opts|
opts.banner = "Usage: #{File.basename($0)}\n" \
@@ -80,10 +79,13 @@ end
if server
+ gem "unicorn"
+ require "pipemaster/server"
+ require "unicorn/launcher"
config = ARGV[0] || "Pipefile"
abort "configuration file #{config} not found" unless File.exist?(config)
options[:config_file] = config
- Dir.chdir File.dirname(config)
+ options[:working_directory] = File.dirname(config)
Unicorn::Launcher.daemonize!(options) if daemonize
Pipemaster.run(options)
View
@@ -1,8 +1,8 @@
-require "unicorn/launcher"
-require "pipemaster/configurator"
-require "pipemaster/worker"
-require "pipemaster/server"
-
module Pipemaster
+ # Version number.
VERSION = Gem::Specification.load(File.expand_path("../pipemaster.gemspec", File.dirname(__FILE__))).version.to_s
+
+ DEFAULT_HOST = "127.0.0.1"
+ DEFAULT_PORT = 7887
+ DEFAULT_LISTEN = "#{DEFAULT_HOST}:#{DEFAULT_PORT}"
end
View
@@ -1,19 +1,43 @@
+require "socket"
+
module Pipemaster
+ # Pipemaster client. Use this to send commands to the Pipemaster server.
+ #
+ # For example:
+ # c = Pipemaster::Client.new
+ # c.request "motd"
+ # puts c.output.string
+ # => "Toilet out of order please use floor below."
+ #
+ # c = Pipemaster::Client.new(9988)
+ # c.input = File.open("image.png")
+ # c.output = File.open("thumbnail.png", "w")
+ # c.request "transform", "thumbnail"
class Client
BUFFER_SIZE = 16 * 1024
+ # Address can be "x.x.x.x:port", ":port" or UNIX socket file name.
def initialize(address)
@address = expand_addr(address || DEFAULT_LISTEN)
end
- attr_accessor :input, :output
+ # Set this to supply an input stream (for commands that read from $stdin).
+ attr_accessor :input
+
+ # Set this to supply an output stream, or read the output (defaults to
+ # StringIO).
+ attr_accessor :output
- def request(*args)
+ # Make a request. First argument is the command name. All other arguments
+ # are optional. Returns the exit code (usually 0). Will raise IOError if
+ # it can't talk to the server, or the server closed the connection
+ # prematurely.
+ def request(command, *args)
# Connect and send arguments.
socket = connect(@address)
socket.sync = true
- args = args.join("\0")
- socket << [args.size].pack("N") << args
+ header = ([command] + args).join("\0")
+ socket << [header.size].pack("N") << header
socket.flush
@output ||= StringIO.new
@@ -46,7 +70,7 @@ def request(*args)
@output.write stdoutbuf if stdoutbuf
stdoutbuf = socket.readpartial(BUFFER_SIZE)
end
- if stdoutbuf
+ if stdoutbuf && stdoutbuf.size > 0
status = stdoutbuf[-1]
@output.write stdoutbuf[0..-2]
return status.ord
@@ -67,6 +67,22 @@ def user(user, group = nil)
Process.euid != uid and Process::UID.change_privilege(uid)
end
+ # Sets the working directory for Pipemaster. Defaults to the location
+ # of the Pipefile. This may be a symlink.
+ def working_directory(path)
+ # just let chdir raise errors
+ path = File.expand_path(path)
+ if config_file &&
+ config_file[0] != ?/ &&
+ ! test(?r, "#{path}/#{config_file}")
+ raise ArgumentError,
+ "pipefile=#{config_file} would not be accessible in" \
+ " working_directory=#{path}"
+ end
+ Dir.chdir(path)
+ Server::START_CTX[:cwd] = ENV["PWD"] = path
+ end
+
# Sets after_fork hook to a given block. This block will be called by
# the worker after forking.
def after_fork(*args, &block)
@@ -202,13 +218,6 @@ def stdout_path(path)
set_path(:stdout_path, path)
end
- # Sets the working directory for Pipemaster. This ensures USR2 will
- # start a new instance of Pipemaster in this directory. This may be
- # a symlink.
- def working_directory(path)
- super
- end
-
private
def preload_app(bool)
View
@@ -1,8 +1,8 @@
-module Pipemaster
+require "unicorn"
+require "pipemaster/worker"
+require "pipemaster/configurator"
- DEFAULT_HOST = "127.0.0.1"
- DEFAULT_PORT = 7887
- DEFAULT_LISTEN = "#{DEFAULT_HOST}:#{DEFAULT_PORT}"
+module Pipemaster
class << self
def run(options = {})
@@ -110,14 +110,14 @@ def reap_all_workers
def load_config!
begin
- logger.info "reloading config_file=#{config.config_file}"
+ logger.info "reloading pipefile=#{config.config_file}"
config[:listeners].replace(init_listeners)
config.reload
config.commit!(self)
Unicorn::Util.reopen_logs
- logger.info "done reloading config_file=#{config.config_file}"
+ logger.info "done reloading pipefile=#{config.config_file}"
rescue => e
- logger.error "error reloading config_file=#{config.config_file}: " \
+ logger.error "error reloading pipefile=#{config.config_file}: " \
"#{e.class} #{e.message}"
end
end
View
@@ -1,6 +1,6 @@
Gem::Specification.new do |spec|
spec.name = "pipemaster"
- spec.version = "0.3.0"
+ spec.version = "0.3.1"
spec.author = "Assaf Arkin"
spec.email = "assaf@labnotes.org"
spec.homepage = "http://labnotes.org"
View
@@ -29,6 +29,7 @@
require 'fileutils'
require 'logger'
require 'pipemaster'
+require 'pipemaster/server'
require 'pipemaster/client'
if ENV['DEBUG']

0 comments on commit 69ab6a8

Please sign in to comment.