New Home: http://beanstalkd.github.com/beaneater/
This fork is no longer maintained.
Beanstalk is a simple, fast work queue. Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. Read the beanstalk protocol for more detail.
Install beanstalk-client as a gem:
gem install beanstalk-client
or add to your Gemfile:
# Gemfile gem 'beanstalk-client'
bundle install to install the dependency.
To interact with a beanstalk queue, first establish a client connection by providing host and port:
@beanstalk = Beanstalk::Pool.new(['10.0.1.5:11300'])
The system has one or more tubes which contain jobs. Each tube consists of a ready queue and a delay queue for jobs.
When a client connects, its watch list is initially just the tube named
If it submits jobs without having sent a
use command, they will live in the tube named
You can specify the tube for a connection with:
@beanstalk.use "some-tube-here" # 'default' if unspecified
Tube names are at most 200 bytes. It specifies the tube to use. If the tube does not exist, it will be created.
A job in beanstalk gets created by a client and includes a 'body' which contains all relevant job metadata. With beanstalk, a job is enqueued into a tube and then later reserved and processed. Here is a picture of the typical job lifecycle:
put reserve delete -----> [READY] ---------> [RESERVED] --------> *poof*
You can put a job onto the beanstalk queue using the
You can also specify additional metadata to control job processing parameters. Specifically,
you can set the
ttr of a particular job:
# defaults are priority 0, delay of 0 and ttr of 120 seconds @beanstalk.put "job-data-here", 1000, 50, 200
priority argument is an integer < 2**32. Jobs with a smaller priority take precedence over jobs with larger priorities.
delay argument is an integer number of seconds to wait before putting the job in the ready queue.
ttr argument is the time to run -- is an integer number of seconds to allow a worker to run this job.
In order to process jobs, the worker first needs to specify which tubes to
watch for new jobs:
@beanstalk = Beanstalk::Pool.new(['10.0.1.5:11300']) @beanstalk.watch('some-tube-name') @beanstalk.watch('some-other-tune')
and perhaps even which tubes to
ignore (including 'default'):
and then we can begin to
reserve jobs. This will find the first job available and
return the job for processing:
job = @beanstalk.reserve # => <Beanstalk::Job> puts job.body # prints 'job-data-here'
You can process each new job as they become available using a loop:
loop do job = @beanstalk.reserve # waits for a job puts job.body # prints "hello" job.delete # remove job after processing end
Beanstalk jobs can also be buried if they fail, rather than deleted:
job = @beanstalk.reserve # ...job fails... job.bury
Burying a job means that the job is pulled out of the queue into a special 'holding' area for later inspection or reuse.
Beanstalk has plenty of commands for introspecting the state of the queues and jobs. These methods include:
# Get overall stats about the job processing that has occured @beanstalk.stats # Get statistical information about the specified job if it exists @beanstalk.job_stats(some_job_id) # Get statistical information about the specified tube if it exists @beanstalk.stats_tube(some_tube_name) # The list-tubes command returns a list of all existing tubes @beanstalk.list_tubes # Returns the tube currently being used by the client @beanstalk.list_tube_used # Returns a list tubes currently being watched by the client @beanstalk.list_tubes_watched
Be sure to check the beanstalk protocol for more details about the stats commands.
There are other resources helpful when learning about beanstalk:
- Isaac Feliu
- Peter Kieltyka
- Martyn Loughran
- Dustin Sallings