Skip to content

Home

mperham edited this page Apr 20, 2011 · 24 revisions
Clone this wiki locally

girl_friday is a Ruby library for performing asynchronous tasks. Often times you don't want to block a web response by performing some task, like sending an email, so you can just use this gem to perform it in the background. It works with any Ruby application, including Rails 3 applications.

Why not use any of the zillions of other async solutions (Resque, dj, etc)? Because girl_friday is easier and more efficient than those solutions: girl_friday runs in your Rails process and uses the actor pattern for safe concurrency. Because it runs in the same process, you don't have to monitor a separate set of processes, deploy a separate codebase, waste hundreds of extra MB in RAM for those processes, etc. See my intro to Actors in Ruby for more detail.

You do need to write thread-safe code. This is not hard to do: the actor pattern means that you get a message and process that message. There is no shared data which requires locks and could lead to deadlock in your application code. Because girl_friday does use Threads under the covers, you need to ensure that your Ruby environment can execute Threads efficiently: today this means JRuby or Rubinius. To be clear: this gem will work but not scale well on Ruby 1.9.

Design

Each queue in girl_friday has two components: a supervisor Actor who listens for errors and manages the queue of work and a pool of worker Actors who know how to process a message. Only the supervisor modifies the internal state of the queue so the queue is inherently threadsafe despite using no locking internally.

How?

The Basics

See Rails for how to use girl_friday in your Rails application.

Advanced Options

Error Handling

If your processor block raises an exception, girl_friday will:

  • kill the worker
  • pass the error to an error handler, which by default will just log the error to $stderr
  • spawn a new worker to replace the one that died

If you are using HoptoadNotifier, girl_friday will pass the exception to HoptoadNotifier.notify_hoptoad(ex) instead of $stderr logging.

You can customize the error handler by passing in a class which has a handle(ex) method:

class MyErrorHandler
  def handle(ex)
    puts ex.message
  end
end

QUEUE = GirlFriday::WorkQueue.new('my_queue', :error_handler => MyErrorHandler) do |msg|
  UserEmail.send_email(msg)
end

Job Persistence

By default, girl_friday just persists jobs to memory but I'm guessing you probably don't want to lose any queued work if you restart your app server instances. To prevent this, girl_friday supports job persistence to a Redis server. You just need to pass in the right options:

QUEUE = GirlFriday::WorkQueue.new('my_queue', :store => GirlFriday::Store::Redis, :store_config => [{ :host => 'hostname', :port => 12345 }]) do |msg|
  UserEmail.send_email(msg)
end

You can remove the store_config parameter if you are using the default configuration ('127.0.0.1:6379').

Clean Shutdown

What about when you want to shut down your application? You can't just kill the process, all the asynchronous work will be left in an unknown state. girl_friday supports a shutdown! method which tells all worker threads to stop picking up new work. They will complete their current jobs and then simply stop. You can continue to push new work onto the queues but that work will simply be pushed to Redis or stored in-memory, it will not be started.

GirlFriday.shutdown!

Note that shutdown! will block until all workers are quiet or timeout after 30 seconds. It will return the number of queues which are still processing, meaning 0 under ideal conditions.

Asynchronous Completion

GirlFriday can call a callback when a message is done processing:

QUEUE.push(:email => @user.email, :name => @user.name) do |result|
  # result = whatever UserEmail.send_email(msg) returned
end

Note: callbacks are incompatible with Redis storage since callbacks can't be marshaled.

Runtime Metrics

You can collect runtime metrics for each queue in your process:

GirlFriday.status

which returns a hash of data for each queue with interesting numbers for you to monitor or graph to your heart's delight:

{"test"=>
  {:pid=>11543,
   :pool_size=>1,
   :ready=>1,
   :busy=>0,
   :backlog=>0,
   :total_queued=>1,
   :total_processed=>1,
   :total_errors=>0,
   :uptime=>0,
   :created_at=>1303092751},
 "image_crawler"=>
  {:pid=>11543,
   :pool_size=>3,
   :ready=>0,
   :busy=>3,
   :backlog=>125,
   :total_queued=>200,
   :total_processed=>72,
   :total_errors=>0,
   :uptime=>0,
   :created_at=>1303092751}}
Something went wrong with that request. Please try again.