Why Beanstalk

Illya has an excellent blog post Scalable Work Queues with Beanstalk and Adam Wiggins posted an excellent comparison.

You can quickly see that beanstalkd is an underrated but incredible project that is extremely well-suited as a job queue. Significantly better suited for this task then Redis or a database. Beanstalk is a simple, and a very fast work queue service rolled into a single binary - it is the memcached of work queues. Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project. PostRank uses beanstalk to reliably process millions of jobs a day.

A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size) because it is an in-memory, event-driven system. Powered by libevent under the hood, it requires zero setup (launch and forget, ala memcached), optional log based persistence, an easily parsed ASCII protocol, and a rich set of tools for job management that go well beyond a simple FIFO work queue.

Beanstalkd supports the following features out of the box:

Feature	Description
Parallelized	Supports multiple work queues created on demand.
Reliable	Beanstalk’s reserve, work, delete cycle ensures reliable processing.
Scheduling	Delay enqueuing jobs by a specified interval to schedule processing later
Fast	Processes thousands of jobs per second without breaking a sweat.
Priorities	Specify priority so important jobs can be processed quickly.
Persistence	Jobs are stored in memory for speed, but logged to disk for safe keeping.
Federation	Horizontal scalability provided through federation by the client.
Error Handling	Bury any job which causes an error for later debugging and inspection.

Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.

In the end, beanstalkd is the ideal job queue while also being ridiculously easy to install and setup.