Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

move all docs into readme

  • Loading branch information...
commit 0eb40ee679428f92cd3c9ae189bff432684fc76f 1 parent 1ed1361
Ryan R. Smith (ace hacker) authored
30 benchmark.rb
View
@@ -1,30 +0,0 @@
-$: << File.expand_path("lib")
-
-require 'benchmark'
-require 'queue_classic'
-
-class StringTest
- def self.length(string)
- string.length
- end
-end
-
-Benchmark.bm(1) do |x|
- n = 10_000
-
- x.report "enqueue" do
- n.times { QC.enqueue("StringTest.length", "foo") }
- end
-
- x.report "forking work" do
- worker = QC::Worker.new
- n.times { worker.fork_and_work }
- end
-
- x.report "work" do
- worker = QC::Worker.new
- n.times { worker.work }
- end
-
-end
-
47 doc/installation.md
View
@@ -1,47 +0,0 @@
-__Rails Compatibility: 2.X and 3.X__
-
-### Gemfile
-
-```ruby
-gem 'queue_classic'
-```
-
-### Rakefile
-
-```ruby
-require 'queue_classic'
-require 'queue_classic/tasks'
-```
-
-### config/initializers/queue_classic.rb
-
-```ruby
-# Optional if you have this set in your shell environment or use Heroku.
-ENV["DATABASE_URL"] = "postgres://username:password@localhost/database_name"
-```
-
-### Database Migration
-
-```ruby
-class CreateJobsTable < ActiveRecord::Migration
-
- def self.up
- create_table :queue_classic_jobs do |t|
- t.text :details
- t.timestamp :locked_at
- end
- add_index :queue_classic_jobs, :id
- end
-
- def self.down
- drop_table :queue_classic_jobs
- end
-
-end
-```
-
-### Load PL/pgSQL Functions
-
-```bash
-$ rake qc:load_functions
-```
40 doc/migrations.md
View
@@ -1,40 +0,0 @@
-# Example migrations
-
-## Sequel
-
-### Jobs Table
-```ruby
-Sequel.migration do
- up do
- create_table :queue_classic_jobs do
- primary_key :id
- String :details
- Time :locked_at
- end
- end
-
- down do
- drop_table :queue_classic_jobs
- end
-end
-```
-
-### Loading the Functions
-```ruby
-def load_qc
- require 'queue_classic'
- ENV['QC_DATABASE_URL'] = self.uri
-end
-
-Sequel.migration do
- up do
- load_qc
- QC::Database.new.load_functions
- end
-
- down do
- load_qc
- QC::Database.new.unload_functions
- end
-end
-```
54 doc/performance.md
View
@@ -1,54 +0,0 @@
-# queue_classic performance
-
-If you look in lib/queue_classic/database.rb you will find the PL/pgSQL function
-lock_head(). This function is responsible for selecting and locking jobs for the
-worker. In this function you will see that it does a trick to set the offset. It
-finds the size of the table and then determines if it should use a random
-offset. At first I wondered if this would be a performance problem, but in
-practice I have not found any such problems. In fact:
-
-On a table with 1,000,000 rows and an index on the ID column, I see the following:
-
-*enqueue*
-
-```sql
-
- queue_classic_test=# EXPLAIN ANALYZE INSERT INTO queue_classic_jobs (details) VALUES ('test');
-
- QUERY PLAN
- ------------------------------------------------------------------------------------------
- Insert (cost=0.00..0.01 rows=1 width=0) (actual time=0.092..0.092 rows=0 loops=1)
- -> Result (cost=0.00..0.01 rows=1 width=0) (actual time=0.029..0.030 rows=1 loops=1)
- Total runtime: 0.174 ms
-
-```
-
-*dequeue*
-
-```sql
-
- queue_classic_test=# EXPLAIN ANALYZE select * from lock_head('queue_classic_jobs');
-
- QUERY PLAN
- -------------------------------------------------------------------------------------------------------------
- Function Scan on lock_head (cost=0.25..10.25 rows=1000 width=44) (actual time=0.824..0.824 rows=1 loops=1)
- Total runtime: 0.842 ms
-
-```
-
-Under the hood, the lock_head function does this:
-
-```sql
-
-queue_classic_test=# EXPLAIN ANALYZE select id from queue_classic_jobs where locked_at IS NULL limit 1 offset 0 for update nowait;
-
-QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------
-Limit (cost=0.00..0.03 rows=1 width=10) (actual time=0.164..0.165 rows=1 loops=1)
- -> LockRows (cost=0.00..29640.02 rows=1010001 width=10) (actual time=0.163..0.163 rows=1 loops=1)
- -> Seq Scan on queue_classic_jobs (cost=0.00..19540.01 rows=1010001 width=10) (actual time=0.149..0.149 rows=1 loops=1)
- Filter: (locked_at IS NULL)
-Total runtime:
-0.227ms
-
-```
44 doc/scheduling.md
View
@@ -1,44 +0,0 @@
-# Scheduling Jobs
-
-Many popular queueing solution provide support for scheduling. Features like
-Redis-Scheduler and the run_at column in DJ are very important to the web
-application developer. While queue_classic does not offer any sort of scheduling
-features, I do not discount the importance of the concept. However, it is my
-belief that a scheduler has no place in a queueing library, to that end I will
-show you how to schedule jobs using queue_classic and the clockwork gem.
-
-## Example
-
-In this example, we are working with a system that needs to compute a sales
-summary at the end of each day. Lets say that we need to compute a summary for
-each sales employee in the system.
-
-Instead of enqueueing jobs with run_at set to 24hour intervals,
-we will define a clock process to enqueue the jobs at a specified
-time on each day. Let us create a file and call it clock.rb:
-
-```ruby
-
- handler {|job| QC.enqueue(job)}
- every(1.day, "SalesSummaryGenerator.build_daily_report", :at => "01:00")
-
-```
-
-To start our scheduler, we will use the clockwork bin:
-
-```bash
- $ clockwork clock.rb
-```
-
-Now each day at 01:00 we will be sending the build_daily_report message to our
-SalesSummaryGenerator class.
-
-I found this abstraction quite powerful and easy to understand. Like
-queue_classic, the clockwork gem is simple to understand and has 0 dependencies.
-In production, I create a heroku process type called clock. This is typically
-what my Procfile looks like:
-
-```bash
-worker: rake jobs:work
-clock: clockwork clock.rb
-```
70 doc/tipsandtricks.md
View
@@ -1,70 +0,0 @@
-# Tips & Tricks
-
-## Running Synchronously for tests
-
-I was tesing some code that started out handling some work in a web request and
-wanted to move that work over to a queue. After completing a red-green-refactor
-I did not want my tests to have to worry about workers or even hit the database.
-
-Turns out its easy to get QueueClassic to just work in a synchronous way with:
-
-```ruby
- def QC.enqueue(function_call, *args)
- eval("#{function_call} *args")
- end
-```
-
-Now you can test QueueClassic as if it was calling your method directly!
-
-
-## Dispatching new jobs to workers without new code
-
-The other day I found myself in a position in which I needed to delete a few
-thousand records. The tough part of this situation is that I needed to ensure
-the ActiveRecord callbacks were made on these objects thus making a simple SQL
-statement unfeasible. Also, I didn't want to wait all day to select and destroy
-these objects. queue_classic to the rescue! (no pun intended)
-
-The API of queue_classic enables you to quickly dispatch jobs to workers. In my
-case I wanted to call `Invoice.destroy(id)` a few thousand times. I fired up a
-heroku console session and executed this line:
-
-```ruby
- Invoice.find(:all, :select => "id", :conditions => "some condition").map {|i| QC.enqueue("Invoice.destroy", i.id) }
-```
-
-With the help of 20 workers I was able to destroy all of these records
-(preserving their callbacks) in a few minutes.
-
-## Enqueueing batches of jobs
-
-I have seen several cases where the application will enqueue jobs in batches. For instance, you may be sending
-1,000 emails out. In this case, it would be foolish to do 1,000 individual transaction. Instead, you want to open
-a new transaction, enqueue all of your jobs and then commit the transaction. This will save tons of time in the
-database.
-
-To achieve this we will create a helper method:
-
-```ruby
-
-def qc_txn
- begin
- QC.database.execute("BEGIN")
- yield
- QC.database.execute("COMMIT")
- rescue Exception
- QC.database.execute("ROLLBACK")
- raise
- end
-end
-```
-
-Now in your application code you can do something like:
-
-```ruby
-qc_txn do
- Account.all.each do |act|
- QC.enqueue("Emailer.send_notice", act.id)
- end
-end
-```
33 doc/upgrade.md
View
@@ -1,33 +0,0 @@
-### 0.2.X to 0.3.X
-
-* Deprecated QC.queue_length in favor of QC.length
-* Locking functions need to be loaded into database via `$ rake qc:load_functions`
-
-
-Also, the default queue is no longer named jobs,
-it is named queue_classic_jobs. Renaming the table is the only change that needs to be made.
-
-```bash
- $ psql your_database
- your_database=# ALTER TABLE jobs RENAME TO queue_classic_jobs;
-```
-
-Or if you are using Rails' Migrations:
-
-```ruby
-class RenameJobsTable < ActiveRecord::Migration
-
- def self.up
- rename_table :jobs, :queue_classic_jobs
- remove_index :jobs, :id
- add_index :queue_classic_jobs, :id
- end
-
- def self.down
- rename_table :queue_classic_jobs, :jobs
- remove_index :queue_classic_jobs, :id
- add_index :jobs, :id
- end
-
-end
-```
54 doc/usage.md
View
@@ -1,54 +0,0 @@
-### Single Queue
-
-You should already have created a table named queue_classic_jobs. This is the default queue.
-You can use the queueing methods on the QC module to interact with the default queue.
-
-```ruby
-QC.enqueue("Class.method","arg1","arg2")
-QC.dequeue
-QC.length
-QC.query("Class.method")
-QC.delete_all
-```
-
-It should be noted that the following enqueue calls do the same thing.
-
-```ruby
-QC.enqueue("Class.method")
-QC::Queue.enqueue("Class.method")
-QC::Queue.new("queue_classic_jobs").enqueue("Class.method")
-```
-
-### Multiple Queues
-
-If you want to create a new queue, you will need to create a new table. The
-table should look identical to the queue_classic_jobs table.
-
-```bash
-$ psql your_database
-psql- CREATE TABLE priority_jobs (id serial, details text, locked_at timestamp);
-psql- CREATE INDEX priority_jobs_id_idx ON priority_jobs (id);
-psql- \q
-```
-
-Once you create a table named "priority_jobs", you will need to create an
-instance of QC::Queue and tell it to attach to your newly created table.
-
-```ruby
-@queue = QC::Queue.new("priority_jobs")
-@queue.enqueue("Class.method", "arg1")
-```
-
-Any method available to the default queue (i.e. QC.enqueue)
-is available to @queue. In fact, both the class and instances
-of the class get their queueing methods from the same module, the AbstractQueue module.
-Look it up in lib/queue_classic/queue.rb for the particulars.
-
-Now, just instruct your worker to attach to your newly created queue.
-
-```bash
-$ rake jobs:work QUEUE="priority_jobs"
-```
-
-For more information regarding the Worker, see the worker page in the docs
-directory.
165 doc/worker.md
View
@@ -1,165 +0,0 @@
-### The Worker
-
-#### General Idea
-
-The worker class (QC::Worker) is designed to be extended via inheritance. Any of
-it's methods should be considered for extension. There are a few in particular
-that act as stubs in hopes that the user will override them. Such methods
-include: `handle_failure() and setup_child()`. See the section near the bottom
-for a detailed descriptor of how to subclass the worker.
-
-#### Algorithm
-
-When we ask the worker to start, it will enter a loop with a stop condition
-dependent upon a method named `running?`. While in the method, the worker will
-attempt to select and lock a job. If it can not on its first attempt, it will
-use an exponential back-off technique to try again.
-
-#### Signals
-
-*INT, TERM* Both of these signals will ensure that the running? method returns
-false. If the worker is waiting -- as it does per the exponential backoff
-technique; then a second signal must be sent.
-
-#### Forking
-
-There are many reasons why you would and would not want your worker to fork.
-An argument against forking may be that you want low latency in your job
-execution. An argument in favor of forking is that your jobs leak memory and do
-all sorts of crazy things, thus warranting the cleanup that fork allows.
-Nevertheless, forking is not enabled by default. To instruct your worker to
-fork, ensure the following shell variable is set:
-
-```bash
-$ export QC_FORK_WORKER='true'
-```
-
-One last note on forking. It is often the case that after Ruby forks a process,
-some sort of setup needs to be done. For instance, you may want to re-establish
-a database connection, or get a new file descriptor. queue_classic's worker
-provides a hook that is called immediately after `Kernel.fork`. To use this hook
-subclass the worker and override `setup_child()`.
-
-#### LISTEN/NOTIFY
-
-The exponential back-off algorithm will require our worker to wait if it does
-not succeed in locking a job. How we wait is something that can vary. PostgreSQL
-has a wonderful feature that we can use to wait intelligently. Processes can LISTEN on a channel and be
-alerted to notifications. queue_classic uses this feature to block until a
-notification is received. If this feature is disabled, the worker will call
-`Kernel.sleep(t)` where t is set by our exponential back-off algorithm. However,
-if we are using LISTEN/NOTIFY then we can enter a type of sleep that can be
-interrupted by a NOTIFY. For example, say we just started to wait for 2 seconds.
-After the first millisecond of waiting, a job was enqueued. With LISTEN/NOTIFY
-enabled, our worker would immediately preempt the wait and attempt to lock the job. This
-allows our worker to be much more responsive. In the case there is no
-notification, the worker will quit waiting after the timeout has expired.
-
-LISTEN/NOTIFY is disabled by default but can be enabled by setting the following shell variable:
-
-```bash
-$ export QC_LISTENING_WORKER='true'
-```
-
-#### Failure
-
-I bet your worker will encounter a job that raises an exception. Queue_classic
-thinks that you should know about this exception by means of you established
-exception tracker. (i.e. Hoptoad, Exceptional) To that end, Queue_classic offers
-a method that you can override. This method will be passed 2 arguments: the
-exception instance and the job. Here are a few examples of things you might want
-to do inside `handle_failure()`.
-
-```ruby
- def handle_failure(job, exception)
- Exceptional.handle(exception, "Background Job Failed" + job.inspect)
-
- HoptoadNotifier.notify(
- :error_class => "Background Job",
- :error_message => "Special Error: #{e.message}",
- :parameters => job.details
- )
-
- # Log to STDOUT (Heroku Logplex listens to stdout)
- puts job.inspect
- puts exception.inspect
- puts exception.backtrace
-
- # Retry the job
- @queue.enqueue(job)
- end
-end
-```
-
-#### Creating a Subclass of QC::Worker
-
-There are many reasons to customize the worker to do exactly what you need.
-QC::Worker was designed to be sub-classed. This section will show a common
-approach to customizing a worker. Somewhere in your project --the lib directory
-works good in a Rails project; you will create a file, call it worker.rb
-
-```ruby
-# lib/worker.rb
-require 'queue_classic'
-
-class MyWorker < QC::Worker
-
- def setup_child
- log("fork establishing database connection")
- ActiveRecord::Base.establish_connection
- end
-
-end
-```
-
-Now that you have created a new worker, you will have to start MyWorker instead
-of QC::Worker. Lets take a look at the different ways to run a worker.
-
-#### Running the Worker
-
-In the installation doc, we showed that including `require 'queue_classic/tasks`
-into your Rakefile would expose `rake jobs:work`. The task defined in
-queue_classic will simply instantiate QC::Worker and call start on that
-instance. This is fine for default setups. However, if you have a customized
-worker or you do not want to use Rake, then the following example will help you
-get your worker started.
-
-For example, lets say that we have a simple Ruby program. We will create a bin
-directory in this project and inside that directory a file named worker.
-
-```ruby
-#!/usr/bin/env ruby
-
-$: << File.expand_path('lib')
-
-require 'queue_classic'
-require 'my_worker'
-
-MyWorker.new.start
-```
-
-Now we can make the file executable and run it using bash.
-
-```bash
-$ chmod +x bin/worker
-$ ./bin/worker
-```
-
-Now we are running our custom worker. The next example will show a similar
-approach but using Rake. In this example, I'll assume we are working with a
-Rails project.
-
-Create a new file lib/tasks/my_worker.rake
-
-```ruby
-require 'queue_classic'
-require 'queue_classic/tasks'
-require 'my_worker'
-# OR you can define MyWorker in this file.
-
-namespace :jobs do
- task :work => :environment do
- MyWorker.new.start
- end
-end
-```
7 lib/queue_classic/worker.rb
View
@@ -72,8 +72,9 @@ def work
if job = lock_job
log("worker locked job=#{job[:id]}")
begin
- call(job)
- log("worker finished job=#{job[:id]}")
+ call(job).tap do
+ log("worker finished job=#{job[:id]}")
+ end
rescue Object => e
log("worker failed job=#{job[:id]} exception=#{e.inspect}")
handle_failure(job, e)
@@ -113,7 +114,7 @@ def call(job)
args = job[:args]
klass = eval(job[:method].split(".").first)
message = job[:method].split(".").last
- klass.send(message, args)
+ klass.send(message, *args)
end
def wait(t)
465 readme.md
View
@@ -13,11 +13,9 @@ queue_classic features:
* JSON encoding for jobs
* Forking workers
* Postgres' rock-solid locking mechanism
-* Fuzzy-FIFO support (1)
+* Fuzzy-FIFO support [academic paper](http://www.cs.tau.ac.il/~shanir/nir-pubs-web/Papers/Lock_Free.pdf)
* Long term support
-1.Theory found here: http://www.cs.tau.ac.il/~shanir/nir-pubs-web/Papers/Lock_Free.pdf
-
## Proven
I wrote queue_classic to solve a production problem. My problem was that I needed a
@@ -44,33 +42,112 @@ Cloudapp processes nearly 14 jobs per second.
I haven't even touched QC since setting it up.
No complaints at all about unreceived emails or non-incrementing view counters.
The best queue is the one you don't have to hand hold.
-```
-- Larry Marburger
+```
-## Quick Start
+## Setup
-See doc/installation.md for Rails instructions
+In addition to installing the rubygem, you will need to prepare your database.
+Database preperation includes creating a table and loading PL/pgSQL functions.
+You can issue the database preperation commands using **PSQL(1)** or place them in a
+database migration.
+
+### Quick Start
```bash
$ createdb queue_classic_test
-$ psql queue_classic_test
-psql- CREATE TABLE queue_classic_jobs (id serial, details text, locked_at timestamp);
+$ psql queue_classic_test -c "CREATE TABLE queue_classic_jobs (id serial, method varchar(255), args text, locked_at timestamp);"
$ export QC_DATABASE_URL="postgres://username:password@localhost/queue_classic_test"
$ gem install queue_classic
-$ ruby -r queue_classic -e "QC::Database.new.load_functions"
+$ ruby -r queue_classic -e "QC::Queries.load_functions"
$ ruby -r queue_classic -e "QC.enqueue('Kernel.puts', 'hello world')"
$ ruby -r queue_classic -e "QC::Worker.new.start"
```
+### Ruby on Rails Setup
+
+**Gemfile**
+
+```ruby
+source :rubygems
+gem "queue_classic", "2.0.0rc1"
+```
+
+**Rakefile**
+
+```ruby
+require "queue_classic"
+require "queue_classic/tasks"
+```
+
+**config/initializers/queue_classic.rb**
+
+```ruby
+# Optional if you have this set in your shell environment or use Heroku.
+ENV["DATABASE_URL"] = "postgres://username:password@localhost/database_name"
+```
+
+**db/migrations/add_queue_classic.rb**
+
+```ruby
+class CreateJobsTable < ActiveRecord::Migration
+
+ def self.up
+ create_table :queue_classic_jobs do |t|
+ t.string :method
+ t.text :args
+ t.timestamp :locked_at
+ end
+ add_index :queue_classic_jobs, :id
+ require "queue_classic"
+ QC::Queries.load_functions
+ end
+
+ def self.down
+ drop_table :queue_classic_jobs
+ require "queue_classic"
+ QC::Queries.drop_functions
+ end
+
+end
+```
+
+### Sequel Setup
+
+**db/migrations/1_add_queue_classic.rb**
+
+```ruby
+Sequel.migration do
+ up do
+ create_table :queue_classic_jobs do
+ primary_key :id
+ String :details
+ Time :locked_at
+ end
+ require "queue_classic"
+ QC::Queries.load_functions
+ end
+
+ down do
+ drop_table :queue_classic_jobs
+ require "queue_classic"
+ QC::Queries.drop_functions
+ end
+end
+```
+
## Configure
```bash
-# Enable logging.
-$VERBOSE
+# Log level.
+# export QC_LOG_LEVEL=`ruby -r "logger" -e "puts Logger::ERROR"`
+$QC_LOG_LEVEL
# Specifies the database that queue_classic will rely upon.
-$QC_DATABASE_URL || $DATABASE_URL
+# queue_classic will try and use QC_DATABASE_URL before it uses DATABASE_URL.
+$QC_DATABASE_URL
+$DATABASE_URL
# Fuzzy-FIFO
# For strict FIFO set to 1. Otherwise, worker will
@@ -79,7 +156,8 @@ $QC_DATABASE_URL || $DATABASE_URL
$QC_TOP_BOUND
# If you want your worker to fork a new
-# child process for each job, set this var to 'true'
+# UNIX process for each job, set this var to 'true'
+#
# Default: false
$QC_FORK_WORKER
@@ -87,21 +165,369 @@ $QC_FORK_WORKER
# if you want high throughput don't use Kernel.sleep
# use LISTEN/NOTIFY sleep. When set to true, the worker's
# sleep will be preempted by insertion into the queue.
+#
# Default: false
$QC_LISTENING_WORKER
# The worker uses an exp backoff algorithm. The base of
-# the exponent is 2. This var determines the max power of the
-# exp.
+# the exponent is 2. This var determines the max power of the exp.
+#
# Default: 5 which implies max sleep time of 2^(5-1) => 16 seconds
$QC_MAX_LOCK_ATTEMPTS
# This var is important for consumers of the queue.
# If you have configured many queues, this var will
# instruct the worker to bind to a particular queue.
-# Default: queue_classic_jobs --which is the default queue table.
+#
+# Default: queue_classic_jobs
$QUEUE
```
+## Usage
+
+Users of queue_classic will be producing jobs (enqueue) or consuming jobs (lock then delete).
+
+### Producer
+
+You certainly don't need the queue_classic rubygem to put a job in the queue.
+
+```bash
+$ psql queue_classic_test -c "INSERT INTO queue_classic_jobs (method, args) VALUES ('Kernel.puts', '[\"hi\"]');"
+```
+
+However, the rubygem will take care of converting your args to JSON and it will also dispatch
+PUB/SUB notifications if the feature is enabled. It will also manage a connection to the database
+that is independent of any other connection you may have in your application. Note: If your
+queue table is in your application's database then your application's process will have 2 connections
+to the database; one for your application and another for queue_classic.
+
+The Ruby API for producing jobs is pretty simple:
+
+```ruby
+# This method has no arguments.
+QC.enqueue("Time.now")
+
+# This method has 1 argument.
+QC.enqueue("Kernel.puts", "hi")
+
+# This method has a hash argument.
+QC.enqueue("MyClass.process", {"credit_card" => "4111"})
+```
+
+The basic idea is that all arguments should be easily encoded to json. OkJson
+is used to encode the arguments, so the arguments can be anything that OkJson can encode.
+
+```ruby
+# Won't work!
+OkJson.encode({:test => "test"})
+
+# OK
+OkJson.encode({"test" => "test"})
+```
+
+To see more information on usage, take a look at the test files in the source code.
+
+### Consumer
+
+Now that you have some jobs in your queue, you probably want to work them.
+Let's find out how... If you are using a Rakefile and have included `queue_classic/tasks`
+then you can enter the following command to start a worker:
+
+#### Rake Task
+
+```bash
+$ bundle exec rake qc:work
+```
+
+#### Bin File
+
+This is the approach that I take when building simple ruby programs and sinatra apps.
+Start by making a bin directory in your project's root directory. Then add a file called
+worker.
+
+**bin/worker**
+
+```ruby
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+trap('INT') {exit}
+trap('TERM') {exit}
+
+require "your_app"
+require "queue_classic"
+worker = QC::Worker.new(table_name, top_bound, fork_worker, listening_worker, max_attempts)
+worker.start
+```
+
+Now that we have seen how to run a worker process, let's take a look at how to customize a worker.
+The class: `QC::Worker` will probably suit most of your needs; however, there are some mechanisms
+that you will want to override. For instance, if you are using a forking worker, you will need to
+open a new database connection in the child process that is doing your work. Also, you may want to
+define how a failed job should behave. The default failed handler will simply print the job to $stdout.
+You can certainly define a failure method that will enqueue the job again, or move it to another table, etc....
+
+#### Sublcass QC::Worker
+
+```ruby
+require "queue_classic"
+
+class MyWorker < QC::Worker
+
+ def handle_failure(job, exception)
+ #retry the job
+ @queue.enque(job[:method], job[:args])
+ end
+
+ def setup_child
+ # the forked proc needs a new db connection
+ ActiveRecord::Base.establish_connection
+ end
+
+end
+```
+
+Notice that we have access to the `@queue` instance variable. Read the tests
+and the worker class for more information.
+
+Now that we have created a new worker, we can run this worker using our bin file:
+
+
+**bin/worker**
+
+```ruby
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+trap('INT') {exit}
+trap('TERM') {exit}
+
+require "your_app"
+require "queue_classic"
+require "my_worker"
+
+worker = MyWorker.new(table_name, top_bound, fork_worker, listening_worker, max_attempts)
+worker.start
+```
+
+#### QC::Worker Details
+
+##### General Idea
+
+The worker class (QC::Worker) is designed to be extended via inheritance. Any of
+it's methods should be considered for extension. There are a few in particular
+that act as stubs in hopes that the user will override them. Such methods
+include: `handle_failure() and setup_child()`. See the section near the bottom
+for a detailed descriptor of how to subclass the worker.
+
+##### Algorithm
+
+When we ask the worker to start, it will enter a loop with a stop condition
+dependent upon a method named `running?`. While in the method, the worker will
+attempt to select and lock a job. If it can not on its first attempt, it will
+use an exponential back-off technique to try again.
+
+##### Signals
+
+*INT, TERM* Both of these signals will ensure that the running? method returns
+false. If the worker is waiting -- as it does per the exponential backoff
+technique; then a second signal must be sent.
+
+##### Forking
+
+There are many reasons why you would and would not want your worker to fork.
+An argument against forking may be that you want low latency in your job
+execution. An argument in favor of forking is that your jobs leak memory and do
+all sorts of crazy things, thus warranting the cleanup that fork allows.
+Nevertheless, forking is not enabled by default. To instruct your worker to
+fork, ensure the following shell variable is set:
+
+```bash
+$ export QC_FORK_WORKER='true'
+```
+
+One last note on forking. It is often the case that after Ruby forks a process,
+some sort of setup needs to be done. For instance, you may want to re-establish
+a database connection, or get a new file descriptor. queue_classic's worker
+provides a hook that is called immediately after `Kernel.fork`. To use this hook
+subclass the worker and override `setup_child()`.
+
+##### LISTEN/NOTIFY
+
+The exponential back-off algorithm will require our worker to wait if it does
+not succeed in locking a job. How we wait is something that can vary. PostgreSQL
+has a wonderful feature that we can use to wait intelligently. Processes can LISTEN on a channel and be
+alerted to notifications. queue_classic uses this feature to block until a
+notification is received. If this feature is disabled, the worker will call
+`Kernel.sleep(t)` where t is set by our exponential back-off algorithm. However,
+if we are using LISTEN/NOTIFY then we can enter a type of sleep that can be
+interrupted by a NOTIFY. For example, say we just started to wait for 2 seconds.
+After the first millisecond of waiting, a job was enqueued. With LISTEN/NOTIFY
+enabled, our worker would immediately preempt the wait and attempt to lock the job. This
+allows our worker to be much more responsive. In the case there is no
+notification, the worker will quit waiting after the timeout has expired.
+
+LISTEN/NOTIFY is disabled by default but can be enabled by setting the following shell variable:
+
+```bash
+$ export QC_LISTENING_WORKER='true'
+```
+
+##### Failure
+
+I bet your worker will encounter a job that raises an exception. Queue_classic
+thinks that you should know about this exception by means of you established
+exception tracker. (i.e. Hoptoad, Exceptional) To that end, Queue_classic offers
+a method that you can override. This method will be passed 2 arguments: the
+exception instance and the job. Here are a few examples of things you might want
+to do inside `handle_failure()`.
+
+## Tips and Tricks
+
+### Running Synchronously for tests
+
+I was tesing some code that started out handling some work in a web request and
+wanted to move that work over to a queue. After completing a red-green-refactor
+I did not want my tests to have to worry about workers or even hit the database.
+
+Turns out its easy to get QueueClassic to just work in a synchronous way with:
+
+```ruby
+ def QC.enqueue(function_call, *args)
+ eval("#{function_call} *args")
+ end
+```
+
+Now you can test QueueClassic as if it was calling your method directly!
+
+
+### Dispatching new jobs to workers without new code
+
+The other day I found myself in a position in which I needed to delete a few
+thousand records. The tough part of this situation is that I needed to ensure
+the ActiveRecord callbacks were made on these objects thus making a simple SQL
+statement unfeasible. Also, I didn't want to wait all day to select and destroy
+these objects. queue_classic to the rescue! (no pun intended)
+
+The API of queue_classic enables you to quickly dispatch jobs to workers. In my
+case I wanted to call `Invoice.destroy(id)` a few thousand times. I fired up a
+heroku console session and executed this line:
+
+```ruby
+ Invoice.find(:all, :select => "id", :conditions => "some condition").map {|i| QC.enqueue("Invoice.destroy", i.id) }
+```
+
+With the help of 20 workers I was able to destroy all of these records
+(preserving their callbacks) in a few minutes.
+
+### Enqueueing batches of jobs
+
+I have seen several cases where the application will enqueue jobs in batches. For instance, you may be sending
+1,000 emails out. In this case, it would be foolish to do 1,000 individual transaction. Instead, you want to open
+a new transaction, enqueue all of your jobs and then commit the transaction. This will save tons of time in the
+database.
+
+To achieve this we will create a helper method:
+
+```ruby
+
+def qc_txn
+ begin
+ QC.database.execute("BEGIN")
+ yield
+ QC.database.execute("COMMIT")
+ rescue Exception
+ QC.database.execute("ROLLBACK")
+ raise
+ end
+end
+```
+
+Now in your application code you can do something like:
+
+```ruby
+qc_txn do
+ Account.all.each do |act|
+ QC.enqueue("Emailer.send_notice", act.id)
+ end
+end
+```
+
+### Scheduling Jobs
+
+Many popular queueing solution provide support for scheduling. Features like
+Redis-Scheduler and the run_at column in DJ are very important to the web
+application developer. While queue_classic does not offer any sort of scheduling
+features, I do not discount the importance of the concept. However, it is my
+belief that a scheduler has no place in a queueing library, to that end I will
+show you how to schedule jobs using queue_classic and the clockwork gem.
+
+#### Example
+
+In this example, we are working with a system that needs to compute a sales
+summary at the end of each day. Lets say that we need to compute a summary for
+each sales employee in the system.
+
+Instead of enqueueing jobs with run_at set to 24hour intervals,
+we will define a clock process to enqueue the jobs at a specified
+time on each day. Let us create a file and call it clock.rb:
+
+```ruby
+handler {|job| QC.enqueue(job)}
+every(1.day, "SalesSummaryGenerator.build_daily_report", :at => "01:00")
+```
+
+To start our scheduler, we will use the clockwork bin:
+
+```bash
+$ clockwork clock.rb
+```
+
+Now each day at 01:00 we will be sending the build_daily_report message to our
+SalesSummaryGenerator class.
+
+I found this abstraction quite powerful and easy to understand. Like
+queue_classic, the clockwork gem is simple to understand and has 0 dependencies.
+In production, I create a heroku process type called clock. This is typically
+what my Procfile looks like:
+
+```
+worker: rake jobs:work
+clock: clockwork clock.rb
+```
+
+## Upgrading From Older Versions
+
+### 0.2.X to 0.3.X
+
+* Deprecated QC.queue_length in favor of QC.length
+* Locking functions need to be loaded into database via `$ rake qc:load_functions`
+
+Also, the default queue is no longer named jobs,
+it is named queue_classic_jobs. Renaming the table is the only change that needs to be made.
+
+```bash
+$ psql your_database -c "ALTER TABLE jobs RENAME TO queue_classic_jobs;"
+```
+
+Or if you are using Rails' Migrations:
+
+```ruby
+class RenameJobsTable < ActiveRecord::Migration
+
+ def self.up
+ rename_table :jobs, :queue_classic_jobs
+ remove_index :jobs, :id
+ add_index :queue_classic_jobs, :id
+ end
+
+ def self.down
+ rename_table :queue_classic_jobs, :jobs
+ remove_index :queue_classic_jobs, :id
+ add_index :jobs, :id
+ end
+
+end
+```
## Hacking on queue_classic
@@ -119,10 +545,3 @@ $ createdb queue_classic_test
$ export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
$ rake
```
-
-## Other Resources
-
-* [Discussion Group](http://groups.google.com/group/queue_classic "discussion group")
-* [Documentation](https://github.com/ryandotsmith/queue_classic/tree/master/doc)
-* [Example Rails App](https://github.com/ryandotsmith/queue_classic_example)
-* [Slide Deck](http://dl.dropbox.com/u/1579953/talks/queue_classic.pdf)
6 test/helper.rb
View
@@ -1,10 +1,10 @@
$: << File.expand_path("lib")
$: << File.expand_path("test")
-ENV['DATABASE_URL'] ||= 'postgres:///queue_classic_test'
+ENV["DATABASE_URL"] ||= "postgres:///queue_classic_test"
-require 'queue_classic'
-require 'minitest/unit'
+require "queue_classic"
+require "minitest/unit"
MiniTest::Unit.autorun
QC::Log.level = Logger::ERROR
38 test/worker_test.rb
View
@@ -1,8 +1,10 @@
require File.expand_path("../helper.rb", __FILE__)
-class TestNotifier
- def self.deliver(args={})
- end
+module TestObject
+ extend self
+ def no_args; return nil; end
+ def one_arg(a); return a; end
+ def two_args(a,b); return [a,b]; end
end
# This not only allows me to test what happens
@@ -25,7 +27,7 @@ def handle_failure(job,e)
class WorkerTest < QCTest
def test_work
- QC.enqueue("TestNotifier.deliver")
+ QC.enqueue("TestObject.no_args")
worker = TestWorker.new("queue_classic_jobs", 1, false, false, 1)
assert_equal(1, QC.count)
worker.work
@@ -34,14 +36,38 @@ def test_work
end
def test_failed_job
- QC.enqueue("TestNotifier.no_method")
+ QC.enqueue("TestObject.not_a_method")
worker = TestWorker.new("queue_classic_jobs", 1, false, false, 1)
worker.work
assert_equal(1, worker.failed_count)
end
+ def test_work_with_no_args
+ QC.enqueue("TestObject.no_args")
+ worker = TestWorker.new("queue_classic_jobs", 1, false, false, 1)
+ r = worker.work
+ assert_nil(r)
+ assert_equal(0, worker.failed_count)
+ end
+
+ def test_work_with_one_arg
+ QC.enqueue("TestObject.one_arg", "1")
+ worker = TestWorker.new("queue_classic_jobs", 1, false, false, 1)
+ r = worker.work
+ assert_equal("1", r)
+ assert_equal(0, worker.failed_count)
+ end
+
+ def test_work_with_two_args
+ QC.enqueue("TestObject.two_args", "1", 2)
+ worker = TestWorker.new("queue_classic_jobs", 1, false, false, 1)
+ r = worker.work
+ assert_equal(["1", 2], r)
+ assert_equal(0, worker.failed_count)
+ end
+
def test_worker_ueses_one_conn
- QC.enqueue("TestNotifier.deliver")
+ QC.enqueue("TestObject.no_args")
worker = TestWorker.new("queue_classic_jobs", 1, false, false, 1)
worker.work
assert_equal(
Please sign in to comment.
Something went wrong with that request. Please try again.