queue_classic is a PostgreSQL-backed queueing library that is focused on concurrent job locking, minimizing database load & providing a simple & intuitive user experience.
- Support for multiple queues with heterogeneous workers
- Utilization of Postgres' PUB/SUB
- JSON encoding for jobs
- Forking workers
- Postgres' rock-solid locking mechanism
- Fuzzy-FIFO support (1)
- Long term support
1.Theory found here: http://www.cs.tau.ac.il/~shanir/nir-pubs-web/Papers/Lock_Free.pdf
I wrote queue_classic to solve a production problem. My problem was that I needed a queueing system that wouldn't fall over should I decide to press it nor should it freak out if I attached 100 workers to it. However, my problem didn't warrant adding an additional service. I was already using PostgreSQL to manage my application's data, why not use PostgreSQL to pass some messages? PostgreSQL was already handling thousands of reads and writes per second anyways. Why not add 35 more reads/writes per second to my established performance metric?
queue_classic handles over 3,000,000 jobs per day. It does this on Heroku's Ronin Database.
See doc/installation.md for Rails instructions
$ createdb queue_classic_test $ psql queue_classic_test psql=# CREATE TABLE queue_classic_jobs (id serial, details text, locked_at timestamp); psql=# CREATE INDEX queue_classic_jobs_id_idx ON queue_classic_jobs (id); $ export QC_DATABASE_URL="postgres://username:password@localhost/queue_classic_test" $ gem install queue_classic $ ruby -r queue_classic -e "QC::Database.new.load_functions" $ ruby -r queue_classic -e "QC.enqueue('Kernel.puts', 'hello world')" $ ruby -r queue_classic -e "QC::Worker.new.start"
# Enable logging. $VERBOSE # Specifies the database that queue_classic will rely upon. $QC_DATABASE_URL || $DATABASE_URL # Fuzzy-FIFO # For strict FIFO set to 1. Otherwise, worker will # attempt to lock a job in this top region. # Default: 9 $QC_TOP_BOUND # If you want your worker to fork a new # child process for each job, set this var to 'true' # Default: false $QC_FORK_WORKER # The worker uses an exp backoff algorithm # if you want high throughput don't use Kernel.sleep # use LISTEN/NOTIFY sleep. When set to true, the worker's # sleep will be preempted by insertion into the queue. # Default: false $QC_LISTENING_WORKER # The worker uses an exp backoff algorithm. The base of # the exponent is 2. This var determines the max power of the # exp. # Default: 5 which implies max sleep time of 2^(5-1) => 16 seconds $QC_MAX_LOCK_ATTEMPTS # This var is important for consumers of the queue. # If you have configured many queues, this var will # instruct the worker to bind to a particular queue. # Default: queue_classic_jobs --which is the default queue table. $QUEUE
Hacking on queue_classic
- Ruby 1.9.2 (tests work in 1.8.7 but compatibility is not guaranteed or supported)
- Postgres ~> 9.0
- Rubygems ** pg ~> 0.11.0 ** json ~> 1.6.1
$ bundle $ createdb queue_classic_test $ export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test" $ rake
If you are adding new features, please document them in the doc directory. Also, once you have the markdown in place, please run: ruby doc/build.rb to make HTML for the docs.