Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
docs are now in the wiki
  • Loading branch information
Ryan R. Smith committed Apr 27, 2011
1 parent 96fbca1 commit a93da63
Show file tree
Hide file tree
Showing 2 changed files with 17 additions and 252 deletions.
6 changes: 3 additions & 3 deletions queue_classic.gemspec
@@ -1,9 +1,9 @@
Gem::Specification.new do |s|
s.name = 'queue_classic'
s.email = 'ryan@heroku.com'
s.version = '0.2.4'
s.date = '2011-02-16'
s.description = "Queue Classic is an alternative queueing library for Ruby apps (Rails, Sinatra, Etc...) Queue Classic features asynchronous job polling, database maintained locks and no ridiculous dependencies. As a matter of fact, Queue Classic only requires the pg and json."
s.version = '0.3.0'
s.date = '2011-04-26'
s.description = "Queue Classic (beta) is a queueing library for Ruby apps (Rails, Sinatra, Etc...) Queue Classic features asynchronous job polling, database maintained locks and no ridiculous dependencies. As a matter of fact, Queue Classic only requires the pg and json."
s.summary = s.description + "(simple)"
s.authors = ["Ryan Smith"]
s.homepage = "http://github.com/ryandotsmith/queue_classic"
Expand Down
263 changes: 14 additions & 249 deletions readme.md
@@ -1,7 +1,7 @@
# Queue Classic
__Beta 0.2.3__
__Beta 0.3.0__

__Queue Classic 0.2.3 is in Beta.__ I have been using this library with 30-150 Heroku workers and have had great results.
__Queue Classic 0.3.0 is in Beta.__ I have been using this library with 30-150 Heroku workers and have had great results.

I am using this in production applications and plan to maintain and support this library for a long time.

Expand All @@ -10,259 +10,24 @@ no ridiculous dependencies. As a matter of fact, Queue Classic only requires the

[Discussion Group](http://groups.google.com/group/queue_classic)

[Wiki](https://github.com/ryandotsmith/queue_classic/wiki)

## Installation

### TL;DR
1. $ gem install queue_classic
2. psql=# CREATE TABLE queue_classic_jobs (id serial, details text, locked_at timestamp);
3. psql=# CREATE INDEX queue_classic_jobs_id_idx ON queue_classic_jobs (id);
4. $ rake qc:load_functions
5. irb: QC.enqueue "Class.method", "arg"
6. $ rake jobs:work

### Upgrade from < 0.2.3 to 0.3.0

1. gem install queue_classic
2. add queue_classic_jobs table to your database
3. QC.enqueue "Class.method", :arg1 => val1
4. rake jobs:work
$ psql your_database
your_database=# ALTER TABLE jobs RENAME TO queue_classic_jobs;

### Dependencies

Postgres version 9

Ruby (gems: pg, json)

### Gem

gem install queue_classic

### Database

Queue Classic needs a database, so make sure that DATABASE_URL points to your database. If you are unsure about whether this var is set, run:
echo $DATABASE_URL
in your shell. If you are using Heroku, this var is set and pointing to your primary database.

Once your Database is set, you will need to add a jobs table. If you are using rails, add a migration with the following tables:

class CreateJobsTable < ActiveRecord::Migration
def self.up
create_table :queue_classic_jobs do |t|
t.text :details
t.timestamp :locked_at
end
add_index :queue_classic_jobs, :id
end

def self.down
drop_table :queue_classic_jobs
end
end
After running this migration, your database should be ready to go. As a sanity check, enqueue a job and then issue a SELECT in the postgres console.

Be sure and add the index to the id column. This will help out the worker if the queue should ever reach an obscene length. It made a huge difference
when running the benchmark.

__script/console__
QC.enqueue "Class.method"
__Terminal__
psql you_database_name
select * from queue_classic_jobs;

You should see the job "Class.method"

### Rakefile

As a convenience, I added a rake task that responds to `rake jobs:work` There are also rake tasks in the `qc` name space.
To get access to these tasks, Add the following to your Rakefile

require 'queue_classic'
require 'queue_classic/tasks'


## Fundamentals

### Enqueue

To place a job onto the queue, you should specify a class and a class method. There are a few ways to enqueue:

##### One Aproach

QC.enqueue('Class.method', :arg1 => 'value1', :arg2 => 'value2')

class Class
def self.method(args)
puts args["arg1"]
end
end

##### Another Aproach

QC.enqueue('Class.method', 'value1', 'value2')

class Class
def self.method(arg1,arg2)
puts arg1
puts arg2
end
end


The job gets stored in the jobs table with a details field set to: `{ job: Class.method, params: {arg1: value1, arg2: value2}}` (as JSON)
Here is a more concrete example of a job implementation using a Rails ActiveRecord Model:

class Invoice < ActiveRecord::Base
def self.process(invoice_id)
invoice = find(invoice_id)
invoice.process!
end

def self.process_all
Invoice.all do |invoice|
QC.enqueue "Invoice.process", invoice.id
end
end
end


### Dequeue

Traditionally, a queue's dequeue operation will remove the item from the queue.
However, Queue Classic will not delete the item from the queue right away; instead,
the workers will lock the job and then the worker will delete the job once it has
finished working it. Queue Classic's greatest strength is it's ability to safely
lock jobs. Unlike other database backed queueing libraries, Queue Classic uses
the database time to lock. This allows you to be more relaxed about the time
synchronization amongst your worker machines.

Queue Classic takes advantage of Postgres' PUB/SUB features to dequeue a job.
Basically there is a channel in which the workers LISTEN. When a new job is added
to the queue, the queue sends NOTIFY messages on the channel. Once a NOTIFY is sent,
each worker races to acquire a lock on a job. A job is awarded to the victor while
the rest go back to wait for another job. This eliminates the need to Sleep & Select.

### The Worker

**Un-handled Exceptions**

The worker calls dequeue and then calls the enqueued method with the supplied arguments.
Once the method terminates, the job is deleted from the queue. In the case that your method
does not terminate, or the worker expectingly dies, Queue Classic will do following:

* Rescue the Exception %
* Call handle_failure(job,exception)
* Delete the job

% - To my knowledge, the only thing that can usurp ensure is a segfault.

**Stopping a worker**

If the worker is in the middle of working a job:
^C => Kill the worker after job is finished.
^C^C => Kill the worker immediately.
If the worker is idle
^C => Kills the worker.

By default, handle_failure will puts the job and the exception. This is not very good and you should override this method. It is simple to do so.
If you are using Queue Classic with Rails, You should:

1. Remove require 'queue_classic/tasks' from Rakefile
2. Create new file in lib/tasks. Call it queue_classic.rb (name is arbitrary)
3. Insert something similar to the following:

#### lib/tasks/queue_classic.rb

require 'queue_classic'

class MyWorker < QC::Worker

def handle_failure(job,exception)
# You can do many things inside of this method. Here are a few examples:

# Log to Exceptional
Exceptional.handle(exception, "Background Job Failed" + job.inspect)

# Log to Hoptoad
HoptoadNotifier.notify(
:error_class => "Background Job",
:error_message => "Special Error: #{e.message}",
:parameters => job.details
)

# Log to STDOUT (Heroku Logplex listens to stdout)
puts job.inspect
puts exception.inspect
puts exception.backtrace

# Retry the job
QC.enqueue(job)

end
end

namespace :jobs do
task :work => :environment do
MyWorker.new.start
end
end

## Performance
I am pleased at the performance of Queue Classic. It ran 3x faster than the DJ. (I have yet to benchmark redis backed queues)

ruby benchmark.rb
user system total real
0.950000 0.620000 1.570000 ( 9.479941)

Hardware: Mac Book Pro 2.8 GHz Intel Core i7. SSD. 4 GB memory.

Software: Ruby 1.9.2-p0, PostgreSQL 9.0.2

It is fast because:

* I wrote my own SQL
* I do not create many Ruby Objects
* I do not call very many methods

## Contribute

* Write Tests (run turn test/ in dir before commit)
* Fork / Pull Request
* Add yourself to the contributors file

Notes on getting tests running:

1. Ensure postgres (>9) is running.
2. $ createdb queue_classic_test
3. $ export DATABASE_URL='postgres://username:password@localhost/queue_classic_test'
4. turn test/

## FAQ

How is this different than DJ?

> __TL;DR__ QC stores job as JSON (better introspection). Postgres manages the time for locking jobs (workers can be out of sync.), No magic (less code), Small footprint (ORM Free).
> __Introspection__ I want the data in the queue to be as simple as possible.
Since we only store the Class, Method and Args, introspection into the queue is quite simple.

> __Locking__ You might have noticed that DJ's worker calls Time.now().
In a cloud environment, this could allow for workers to be confused about the status
of a job. Classic Queue locks a job using Postgres' TIMESTAMP function.

> __Magic__ I find disdain for methods on my objects that have nothing to do
with the purpose of the object. Methods like "should" and "delay" are quite distasteful
and obscure what is actually going on. If you use TestUnit for this reason,
you might like Queue Classic. Anyway, I think the fundamental concept of a message
queue is not that difficult to grasp; therefore, I have taken the time to make
Queue Classic as transparent as possible.

> __Footprint__ You don't need ActiveRecord or any other ORM to find the head or
add to the tail. Take a look at the DurableArray class to see the SQL Queue Classic employees.

Why doesn't your queue retry failed jobs?

> I believe the Class method should handle any sort of exception. Also, I think
that the model you are working on should know about it's state. For instance, if you are
creating jobs for the emailing of newsletters; put a emailed_at column on your newsletter model
and then right before the job quits, touch the emailed_at column. That being said, you can do whatever you
want in handle_failure. I will not decide what is best for your application.

Can I use this library with 50 Heroku Workers?

> Yes. Why not 100? Make sure your database can handle the connections. Each workers will consume a database connection.
Is Queue Classic ready for production?

> I started this project on 1/24/2011. I have been using this in production for some high-traffic apps at Heroku since 2/24/2011.

0 comments on commit a93da63

Please sign in to comment.