Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
bin * Added: Ping command: Mar 1, 2010


You pipe from here, to a process that runs over there. Pipemaster forks, redirects and runs your command.


I've got a short task I want to perform. To validate an incoming email, parse the message and store the results in the database(*). I setup Postfix to pipe incoming emails into a Ruby script.

Ruby processes are fairly cheap, until you get into loading the mail library, the database library, the ORM, the application logic, the … you get the picture.

I use Pipemaster to fire up the main process once, and the Pipemaster client to run commands on that server. Still have to make sure the code is light and fast, but I did eliminate the significant initialization overhead.

As you can guess, Pipemaster supports piping input and output streams, and uses forking (sorry Windows; for JRuby see Nailgun).

  • Processing emails as they come allows the application to reject unauthorized senders immediately by replying with an SMTP code. The alternative, accepting the email and later on sending a bounce, leads to backscatter (see

Using Pipemaster

Step 1: Create a Pipemaster file. For example:

command :echo do |*args|
  first, rest = $
  $stdout << [first, args, rest].flatten.join(" ")

Step 2: Fire up the Pipemaster server:

$ pipemaster
I, [2010-02-18T12:57:57.739230 #5460]  INFO -- : master process ready
I, [2010-02-18T12:57:57.739606 #5460]  INFO -- : listening on addr= fd=3

Step 3: For a new shell, execute a command:

$ echo "Stand down!" | pipe echo upside
Stand upside down!

Pipemaster, Resque and Rails

This example uses Resque to queue and process tasks asynchronously, where the tasks are part of a larger Rails application (e.g. using ActiveRecord models, ActiveMailer).

This Pipefile loads the Rails application once during setup. It starts one Resque worker than polls for new jobs every 5 seconds.

#!/usr/bin/env ruby -S pipemaster
user  "nobody"

require "syslog_logger"
syslog ="pipemaster")
class << syslog ; def close ; end ; end
logger syslog

setup do
  # Load RAILS.  Diz will take a while.
  require File.dirname(__FILE__) + '/config/environment'
after_fork do |server, worker|

# Resque
background :resque do
  resque ="*")
  resque.verbose = true
  trap(:QUIT) { resque.shutdown } # graceful

Tips && tricks

Add this at the top of your Pipefile for Ruby syntax highlighting:

#!ruby -S pipemaster


Pipemaster is copyright of Assaf Arkin. It is heavily based on the awesome Unicorn Web server and therefore uses the same license.

Unicorn is copyright 2009 by all contributors (see logs in git). It is based on Mongrel and carries the same license.

Mongrel is copyright 2007 Zed A. Shaw and contributors. It is licensed under the Ruby license and the GPL2. See the included LICENSE file for details.

Pipemaster is 100% Free Software.