Beanstalk + Tokyo Tyrant = Edamame, a fast persistent distributed priority job queue
- fast, scalable, lightweight and distributed
- persistent and recoverable
- scalable up to your memory limits
- queryable and enumerable jobs
- named jobs
- reasonably-good availability.
Like beanstalk, it is a job queue, not just a message queue:
- priority job scheduling, not just FIFO
- Supports multiple queues (‘tubes’)
- reliable scheduling: jobs that time out are re-assigned
It includes a few nifty toys:
- Scripts for God to monitor and restart the daemons
- Command-line management scripts to load. enumerate, empty, and show stats for the db+queue
- The start of a lightweight web frontend in Sinatra.
The bulk of the documentation is at http://mrflip.github.com/edamame Go there instead.
Send Edamame questions to the Infinite Monkeywrench mailing list
Requirements and Installation
Get the code
$ git clone git://github.com/mrflip/edamame
A gem is available from gemcutter:
$ sudo gem install edamame --source=http://gemcutter.org
(don’t use the gems.github.com version — it’s way out of date.)
Get the Dependencies
- beanstalkd 1.3, libevent 1.4, and beanstalk-client
- Tokyo Tyrant, Tokyo Tyrant Ruby libs, Tokyo Cabinet, and Tokyo Cabinet Ruby libs
- Gems: wukong and monkeyshines
See the Detailed install instructions (it also has hints about installing Tokyo*, Beanstalkd and friends.
Weaknesses? Mainly that it will make an Erlang’er cry for its lack of concurrency correctness. Its goal is to work pretty well and to recover gracefully, but its design limits .
- We store jobs in two places: the central DB and the distributed queue.
- As always, your jobs must either be idempotent, or harmless if re-run: a job could start and do some or all of its job — but lose contact with the queue, causing the job to be re-run. This is inherent in beanstalkd (and most comparable solutions), not just edamame.
- Although God will watch the daemons, it won’t repopulate the queue or restart a worker that fails.
- Restarting is still manual: you have to run
bin/edamame-syncto reload the queue from the database
- The sinatra queue viewer doesn’t work at the moment.
There’s a fuller set of docs at http://mrflip.github.com/edamame
- Origin of the name edamame
- This library was written to support the Monkeyshines distributed API scraper.
- Tokyo Tyrant:
- God process monitoring framework