Skip to content

Why Beanstalk

nesquena edited this page Apr 19, 2013 · 5 revisions

Illya has an excellent blog post Scalable Work Queues with Beanstalk and Adam Wiggins posted an excellent comparison.

You can quickly see that beanstalkd is an underrated but incredible project that is extremely well-suited as a job queue. Significantly better suited for this task then Redis or a database. Beanstalk is a simple, and a very fast work queue service rolled into a single binary - it is the memcached of work queues. Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project. PostRank uses beanstalk to reliably process millions of jobs a day.

A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size) because it is an in-memory, event-driven system. Powered by libevent under the hood, it requires zero setup (launch and forget, ala memcached), optional log based persistence, an easily parsed ASCII protocol, and a rich set of tools for job management that go well beyond a simple FIFO work queue.

Beanstalkd supports the following features out of the box:

Feature Description
Parallelized Supports multiple work queues created on demand.
Reliable Beanstalk’s reserve, work, delete cycle ensures reliable processing.
Scheduling Delay enqueuing jobs by a specified interval to schedule processing later
Fast Processes thousands of jobs per second without breaking a sweat.
Priorities Specify priority so important jobs can be processed quickly.
Persistence Jobs are stored in memory for speed, but logged to disk for safe keeping.
Federation Horizontal scalability provided through federation by the client.
Error Handling Bury any job which causes an error for later debugging and inspection.

Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.

In the end, beanstalkd is the ideal job queue while also being ridiculously easy to install and setup.

Something went wrong with that request. Please try again.