Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 909 lines (597 sloc) 24.886 kb
7a775a9 @defunkt Document signals before we forget
defunkt authored
1 Resque
2 ======
b4b2b29 @defunkt test
defunkt authored
3
7ca0bf3 @mrduncan Remove trailing whitespace from readme
mrduncan authored
4 Resque (pronounced like "rescue") is a Redis-backed library for creating
5 background jobs, placing those jobs on multiple queues, and processing
52b4a28 @defunkt It's pronounced 'rescue'
defunkt authored
6 them later.
7aed61a @defunkt begin the readme
defunkt authored
7
94d5ff9 @defunkt readme
defunkt authored
8 Background jobs can be any Ruby class or module that responds to
9 `perform`. Your existing classes can easily be converted to background
10 jobs or you can create new classes specifically to do work. Or, you
11 can do both.
7aed61a @defunkt begin the readme
defunkt authored
12
c70d454 @chrislerum Fix grammar in README
chrislerum authored
13 Resque is heavily inspired by DelayedJob (which rocks) and comprises
14 three parts:
94d5ff9 @defunkt readme
defunkt authored
15
3a783ee @defunkt simple README tweaks
defunkt authored
16 1. A Ruby library for creating, querying, and processing jobs
2c5a113 @defunkt readme updates
defunkt authored
17 2. A Rake task for starting a worker which processes jobs
94d5ff9 @defunkt readme
defunkt authored
18 3. A Sinatra app for monitoring queues, jobs, and workers.
7aed61a @defunkt begin the readme
defunkt authored
19
a8e2e26 @defunkt more readme, but some lies
defunkt authored
20 Resque workers can be distributed between multiple machines,
2be99f6 @jgeiger fix more spelling errors
jgeiger authored
21 support priorities, are resilient to memory bloat / "leaks," are
935a100 @defunkt more the readme
defunkt authored
22 optimized for REE (but work on MRI and JRuby), tell you what they're
23 doing, and expect failure.
24
2c5a113 @defunkt readme updates
defunkt authored
25 Resque queues are persistent; support constant time, atomic push and
26 pop (thanks to Redis); provide visibility into their contents; and
94d5ff9 @defunkt readme
defunkt authored
27 store jobs as simple JSON packages.
935a100 @defunkt more the readme
defunkt authored
28
29 The Resque frontend tells you what workers are doing, what workers are
30 not doing, what queues you're using, what's in those queues, provides
31 general usage stats, and helps you track failures.
32
33
f2e1ee3 @defunkt remove the blog post from the README
defunkt authored
34 The Blog Post
35 -------------
7aed61a @defunkt begin the readme
defunkt authored
36
f2e1ee3 @defunkt remove the blog post from the README
defunkt authored
37 For the backstory, philosophy, and history of Resque's beginnings,
38 please see [the blog post][0].
935a100 @defunkt more the readme
defunkt authored
39
2c5a113 @defunkt readme updates
defunkt authored
40
a8e2e26 @defunkt more readme, but some lies
defunkt authored
41 Overview
42 --------
43
44 Resque allows you to create jobs and place them on a queue, then,
45 later, pull those jobs off the queue and process them.
46
94d5ff9 @defunkt readme
defunkt authored
47 Resque jobs are Ruby classes (or modules) which respond to the
48 `perform` method. Here's an example:
a8e2e26 @defunkt more readme, but some lies
defunkt authored
49
f116bac @defunkt whitespace
defunkt authored
50
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
51 ``` ruby
52 class Archive
53 @queue = :file_serve
54
55 def self.perform(repo_id, branch = 'master')
56 repo = Repository.find(repo_id)
57 repo.create_archive(branch)
58 end
59 end
60 ```
a8e2e26 @defunkt more readme, but some lies
defunkt authored
61
94d5ff9 @defunkt readme
defunkt authored
62 The `@queue` class instance variable determines which queue `Archive`
63 jobs will be placed in. Queues are arbitrary and created on the fly -
64 you can name them whatever you want and have as many as you want.
65
66 To place an `Archive` job on the `file_serve` queue, we might add this
67 to our application's pre-existing `Repository` class:
a8e2e26 @defunkt more readme, but some lies
defunkt authored
68
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
69 ``` ruby
70 class Repository
71 def async_create_archive(branch)
72 Resque.enqueue(Archive, self.id, branch)
73 end
74 end
75 ```
a8e2e26 @defunkt more readme, but some lies
defunkt authored
76
77 Now when we call `repo.async_create_archive('masterbrew')` in our
78 application, a job will be created and placed on the `file_serve`
79 queue.
80
94d5ff9 @defunkt readme
defunkt authored
81 Later, a worker will run something like this code to process the job:
f116bac @defunkt whitespace
defunkt authored
82
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
83 ``` ruby
84 klass, args = Resque.reserve(:file_serve)
85 klass.perform(*args) if klass.respond_to? :perform
86 ```
94d5ff9 @defunkt readme
defunkt authored
87
88 Which translates to:
f116bac @defunkt whitespace
defunkt authored
89
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
90 ``` ruby
91 Archive.perform(44, 'masterbrew')
92 ```
4e02287 @defunkt take a break
defunkt authored
93
a8e2e26 @defunkt more readme, but some lies
defunkt authored
94 Let's start a worker to run `file_serve` jobs:
95
96 $ cd app_root
97 $ QUEUE=file_serve rake resque:work
98
99 This starts one Resque worker and tells it to work off the
100 `file_serve` queue. As soon as it's ready it'll try to run the
101 `Resque.reserve` code snippet above and process jobs until it can't
102 find any more, at which point it will sleep for a small period and
103 repeatedly poll the queue for more jobs.
b4b2b29 @defunkt test
defunkt authored
104
f2e1ee3 @defunkt remove the blog post from the README
defunkt authored
105 Workers can be given multiple queues (a "queue list") and run on
106 multiple machines. In fact they can be run anywhere with network
107 access to the Redis server.
108
3a783ee @defunkt simple README tweaks
defunkt authored
109
2c5a113 @defunkt readme updates
defunkt authored
110 Jobs
111 ----
112
113 What should you run in the background? Anything that takes any time at
114 all. Slow INSERT statements, disk manipulating, data processing, etc.
115
116 At GitHub we use Resque to process the following types of jobs:
117
118 * Warming caches
119 * Counting disk usage
120 * Building tarballs
121 * Building Rubygems
122 * Firing off web hooks
123 * Creating events in the db and pre-caching them
124 * Building graphs
125 * Deleting users
126 * Updating our search index
127
128 As of writing we have about 35 different types of background jobs.
129
130 Keep in mind that you don't need a web app to use Resque - we just
131 mention "foreground" and "background" because they make conceptual
132 sense. You could easily be spidering sites and sticking data which
133 needs to be crunched later into a queue.
134
135
136 ### Persistence
137
138 Jobs are persisted to queues as JSON objects. Let's take our `Archive`
139 example from above. We'll run the following code to create a job:
140
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
141 ``` ruby
142 repo = Repository.find(44)
143 repo.async_create_archive('masterbrew')
144 ```
2c5a113 @defunkt readme updates
defunkt authored
145
146 The following JSON will be stored in the `file_serve` queue:
147
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
148 ``` javascript
149 {
150 'class': 'Archive',
151 'args': [ 44, 'masterbrew' ]
152 }
153 ```
2c5a113 @defunkt readme updates
defunkt authored
154
155 Because of this your jobs must only accept arguments that can be JSON encoded.
156
157 So instead of doing this:
158
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
159 ``` ruby
160 Resque.enqueue(Archive, self, branch)
161 ```
2c5a113 @defunkt readme updates
defunkt authored
162
163 do this:
164
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
165 ``` ruby
166 Resque.enqueue(Archive, self.id, branch)
167 ```
f116bac @defunkt whitespace
defunkt authored
168
2c5a113 @defunkt readme updates
defunkt authored
169 This is why our above example (and all the examples in `examples/`)
170 uses object IDs instead of passing around the objects.
171
2be99f6 @jgeiger fix more spelling errors
jgeiger authored
172 While this is less convenient than just sticking a marshaled object
2c5a113 @defunkt readme updates
defunkt authored
173 in the database, it gives you a slight advantage: your jobs will be
174 run against the most recent version of an object because they need to
175 pull from the DB or cache.
176
2be99f6 @jgeiger fix more spelling errors
jgeiger authored
177 If your jobs were run against marshaled objects, they could
2c5a113 @defunkt readme updates
defunkt authored
178 potentially be operating on a stale record with out-of-date information.
179
180
181 ### send_later / async
182
183 Want something like DelayedJob's `send_later` or the ability to use
184 instance methods instead of just methods for jobs? See the `examples/`
185 directory for goodies.
186
187 We plan to provide first class `async` support in a future release.
188
3a783ee @defunkt simple README tweaks
defunkt authored
189
2c5a113 @defunkt readme updates
defunkt authored
190 ### Failure
3a783ee @defunkt simple README tweaks
defunkt authored
191
2c5a113 @defunkt readme updates
defunkt authored
192 If a job raises an exception, it is logged and handed off to the
193 `Resque::Failure` module. Failures are logged either locally in Redis
194 or using some different backend.
3a783ee @defunkt simple README tweaks
defunkt authored
195
986d8ec @defunkt lies
defunkt authored
196 For example, Resque ships with Hoptoad support.
2c5a113 @defunkt readme updates
defunkt authored
197
198 Keep this in mind when writing your jobs: you may want to throw
199 exceptions you would not normally throw in order to assist debugging.
94d5ff9 @defunkt readme
defunkt authored
200
f2e1ee3 @defunkt remove the blog post from the README
defunkt authored
201
94d5ff9 @defunkt readme
defunkt authored
202 Workers
7a775a9 @defunkt Document signals before we forget
defunkt authored
203 -------
c1210da @defunkt test
defunkt authored
204
2b07897 @jgeiger fix spelling issue
jgeiger authored
205 Resque workers are rake tasks that run forever. They basically do this:
2c5a113 @defunkt readme updates
defunkt authored
206
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
207 ``` ruby
208 start
209 loop do
210 if job = reserve
211 job.process
212 else
50fee9b @jeznet Mentioned and explained the INTERVAL option in the README.
jeznet authored
213 sleep 5 # Polling frequency = 5
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
214 end
215 end
216 shutdown
217 ```
2c5a113 @defunkt readme updates
defunkt authored
218
219 Starting a worker is simple. Here's our example from earlier:
220
221 $ QUEUE=file_serve rake resque:work
222
223 By default Resque won't know about your application's
224 environment. That is, it won't be able to find and run your jobs - it
225 needs to load your application into memory.
226
227 If we've installed Resque as a Rails plugin, we might run this command
228 from our RAILS_ROOT:
229
230 $ QUEUE=file_serve rake environment resque:work
231
232 This will load the environment before starting a worker. Alternately
233 we can define a `resque:setup` task with a dependency on the
234 `environment` rake task:
235
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
236 ``` ruby
237 task "resque:setup" => :environment
238 ```
2c5a113 @defunkt readme updates
defunkt authored
239
6b96c20 @defunkt installation dreams
defunkt authored
240 GitHub's setup task looks like this:
241
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
242 ``` ruby
243 task "resque:setup" => :environment do
244 Grit::Git.git_timeout = 10.minutes
245 end
246 ```
6b96c20 @defunkt installation dreams
defunkt authored
247
248 We don't want the `git_timeout` as high as 10 minutes in our web app,
249 but in the Resque workers it's fine.
250
2c5a113 @defunkt readme updates
defunkt authored
251
9b855ba @defunkt MySQL server has gone away
defunkt authored
252 ### Logging
931fcbd @defunkt basic note on logging
defunkt authored
253
254 Workers support basic logging to STDOUT. If you start them with the
255 `VERBOSE` env variable set, they will print basic debugging
256 information. You can also set the `VVERBOSE` (very verbose) env
257 variable.
258
259 $ VVERBOSE=1 QUEUE=file_serve rake environment resque:work
260
5765fe1 @christiannelson Write out worker pid to PIDFILE if specified.
christiannelson authored
261 ### Process IDs (PIDs)
262
263 There are scenarios where it's helpful to record the PID of a resque
264 worker process. Use the PIDFILE option for easy access to the PID:
265
266 $ PIDFILE=./resque.pid QUEUE=file_serve rake environment resque:work
267
39d51cb @cleishm Added documentation of daemonization
cleishm authored
268 ### Running in the background
269
dcfbe36 @cleishm Add a notes regarding ruby >= 1.9 requirement for BACKGROUND
cleishm authored
270 (Only supported with ruby >= 1.9). There are scenarios where it's helpful for
271 the resque worker to run itself in the background (usually in combination with
272 PIDFILE). Use the BACKGROUND option so that rake will return as soon as the
273 worker is started.
39d51cb @cleishm Added documentation of daemonization
cleishm authored
274
275 $ PIDFILE=./resque.pid BACKGROUND=yes QUEUE=file_serve \
276 rake environment resque:work
931fcbd @defunkt basic note on logging
defunkt authored
277
50fee9b @jeznet Mentioned and explained the INTERVAL option in the README.
jeznet authored
278 ### Polling frequency
279
280 You can pass an INTERVAL option which is a float representing the polling frequency.
281 The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
282
283 $ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
284
5e34fd1 @defunkt queue lists
defunkt authored
285 ### Priorities and Queue Lists
2c5a113 @defunkt readme updates
defunkt authored
286
5e34fd1 @defunkt queue lists
defunkt authored
287 Resque doesn't support numeric priorities but instead uses the order
288 of queues you give it. We call this list of queues the "queue list."
289
290 Let's say we add a `warm_cache` queue in addition to our `file_serve`
291 queue. We'd now start a worker like so:
292
293 $ QUEUES=file_serve,warm_cache rake resque:work
294
295 When the worker looks for new jobs, it will first check
296 `file_serve`. If it finds a job, it'll process it then check
297 `file_serve` again. It will keep checking `file_serve` until no more
298 jobs are available. At that point, it will check `warm_cache`. If it
299 finds a job it'll process it then check `file_serve` (repeating the
300 whole process).
301
302 In this way you can prioritize certain queues. At GitHub we start our
303 workers with something like this:
304
305 $ QUEUES=critical,archive,high,low rake resque:work
306
307 Notice the `archive` queue - it is specialized and in our future
308 architecture will only be run from a single machine.
309
310 At that point we'll start workers on our generalized background
311 machines with this command:
312
313 $ QUEUES=critical,high,low rake resque:work
314
315 And workers on our specialized archive machine with this command:
316
317 $ QUEUE=archive rake resque:work
318
319
320 ### Running All Queues
321
322 If you want your workers to work off of every queue, including new
323 queues created on the fly, you can use a splat:
324
325 $ QUEUE=* rake resque:work
326
f116bac @defunkt whitespace
defunkt authored
327 Queues will be processed in alphabetical order.
328
2c5a113 @defunkt readme updates
defunkt authored
329
27dbbdb @defunkt Added `rake resque:workers` task for starting multiple workers.
defunkt authored
330 ### Running Multiple Workers
331
332 At GitHub we use god to start and stop multiple workers. A sample god
333 configuration file is included under `examples/god`. We recommend this
334 method.
335
336 If you'd like to run multiple workers in development mode, you can do
337 so using the `resque:workers` rake task:
338
339 $ COUNT=5 QUEUE=* rake resque:workers
340
341 This will spawn five Resque workers, each in its own thread. Hitting
342 ctrl-c should be sufficient to stop them all.
343
344
2c5a113 @defunkt readme updates
defunkt authored
345 ### Forking
346
3b9fe26 @defunkt talk about forking
defunkt authored
347 On certain platforms, when a Resque worker reserves a job it
348 immediately forks a child process. The child processes the job then
349 exits. When the child has exited successfully, the worker reserves
350 another job and repeats the process.
351
352 Why?
353
354 Because Resque assumes chaos.
355
356 Resque assumes your background workers will lock up, run too long, or
357 have unwanted memory growth.
358
359 If Resque workers processed jobs themselves, it'd be hard to whip them
360 into shape. Let's say one is using too much memory: you send it a
361 signal that says "shutdown after you finish processing the current
362 job," and it does so. It then starts up again - loading your entire
363 application environment. This adds useless CPU cycles and causes a
364 delay in queue processing.
365
366 Plus, what if it's using too much memory and has stopped responding to
367 signals?
368
369 Thanks to Resque's parent / child architecture, jobs that use too much memory
370 release that memory upon completion. No unwanted growth.
371
372 And what if a job is running too long? You'd need to `kill -9` it then
373 start the worker again. With Resque's parent / child architecture you
374 can tell the parent to forcefully kill the child then immediately
375 start processing more jobs. No startup delay or wasted cycles.
376
377 The parent / child architecture helps us keep tabs on what workers are
378 doing, too. By eliminating the need to `kill -9` workers we can have
379 parents remove themselves from the global listing of workers. If we
380 just ruthlessly killed workers, we'd need a separate watchdog process
381 to add and remove them to the global listing - which becomes
382 complicated.
383
384 Workers instead handle their own state.
2c5a113 @defunkt readme updates
defunkt authored
385
386
33f0fd4 @defunkt start parent / child section
defunkt authored
387 ### Parents and Children
388
389 Here's a parent / child pair doing some work:
390
1c45ca1 @defunkt parents and children
defunkt authored
391 $ ps -e -o pid,command | grep [r]esque
f116bac @defunkt whitespace
defunkt authored
392 92099 resque: Forked 92102 at 1253142769
393 92102 resque: Processing file_serve since 1253142769
33f0fd4 @defunkt start parent / child section
defunkt authored
394
1c45ca1 @defunkt parents and children
defunkt authored
395 You can clearly see that process 92099 forked 92102, which has been
396 working since 1253142769.
397
398 (By advertising the time they began processing you can easily use monit
399 or god to kill stale workers.)
400
401 When a parent process is idle, it lets you know what queues it is
402 waiting for work on:
403
404 $ ps -e -o pid,command | grep [r]esque
405 92099 resque: Waiting for file_serve,warm_cache
33f0fd4 @defunkt start parent / child section
defunkt authored
406
407
94d5ff9 @defunkt readme
defunkt authored
408 ### Signals
409
2c5a113 @defunkt readme updates
defunkt authored
410 Resque workers respond to a few different signals:
411
3073c49 @defunkt use QUIT, not HUP, to gracefully shutdown
defunkt authored
412 * `QUIT` - Wait for child to finish processing then exit
4f61f3e @defunkt fix readme formatting
defunkt authored
413 * `TERM` / `INT` - Immediately kill child then exit
2c5a113 @defunkt readme updates
defunkt authored
414 * `USR1` - Immediately kill child but don't exit
1d033dc @mpalmer Handle USR2/CONT signals, to pause and unpause workers
mpalmer authored
415 * `USR2` - Don't start to process any new jobs
416 * `CONT` - Start to process new jobs again after a USR2
2c5a113 @defunkt readme updates
defunkt authored
417
418 If you want to gracefully shutdown a Resque worker, use `QUIT`.
419
420 If you want to kill a stale or stuck child, use `USR1`. Processing
38b83cb @defunkt If USR1 is sent and no child is found, shutdown.
defunkt authored
421 will continue as normal unless the child was not found. In that case
422 Resque assumes the parent process is in a bad state and shuts down.
2c5a113 @defunkt readme updates
defunkt authored
423
424 If you want to kill a stale or stuck child and shutdown, use `TERM`
425
1d033dc @mpalmer Handle USR2/CONT signals, to pause and unpause workers
mpalmer authored
426 If you want to stop processing jobs, but want to leave the worker running
427 (for example, to temporarily alleviate load), use `USR2` to stop processing,
428 then `CONT` to start it again.
429
9b855ba @defunkt MySQL server has gone away
defunkt authored
430 ### Mysql::Error: MySQL server has gone away
431
432 If your workers remain idle for too long they may lose their MySQL
433 connection. If that happens we recommend using [this
434 Gist](http://gist.github.com/238999).
435
94d5ff9 @defunkt readme
defunkt authored
436
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
437 The Front End
438 -------------
439
440 Resque comes with a Sinatra-based front end for seeing what's up with
f116bac @defunkt whitespace
defunkt authored
441 your queue.
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
442
157926e @defunkt Even better screenshot in README
defunkt authored
443 ![The Front End](https://img.skitch.com/20110528-pc67a8qsfapgjxf5gagxd92fcu.png)
9d6de6a @defunkt frontend sneak preview
defunkt authored
444
9b855ba @defunkt MySQL server has gone away
defunkt authored
445 ### Standalone
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
446
447 If you've installed Resque as a gem running the front end standalone is easy:
448
449 $ resque-web
450
451 It's a thin layer around `rackup` so it's configurable as well:
452
453 $ resque-web -p 8282
454
455 If you have a Resque config file you want evaluated just pass it to
456 the script as the final argument:
457
458 $ resque-web -p 8282 rails_root/config/initializers/resque.rb
f116bac @defunkt whitespace
defunkt authored
459
e7a3f5a @defunkt Mention -N in the readme
defunkt authored
460 You can also set the namespace directly using `resque-web`:
461
462 $ resque-web -p 8282 -N myapp
463
df89630 @gjastrab Adds -r flag to resque-web for redis connection
gjastrab authored
464 or set the Redis connection string if you need to do something like select a different database:
465
466 $ resque-web -p 8282 -r localhost:6379:2
467
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
468 ### Passenger
469
470 Using Passenger? Resque ships with a `config.ru` you can use. See
471 Phusion's guide:
472
6a3a283 @sgonyea Fixed links for deploying rack apps
sgonyea authored
473 Apache: <http://www.modrails.com/documentation/Users%20guide%20Apache.html#_deploying_a_rack_based_ruby_application>
474 Nginx: <http://www.modrails.com/documentation/Users%20guide%20Nginx.html#deploying_a_rack_app>
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
475
476 ### Rack::URLMap
477
478 If you want to load Resque on a subpath, possibly alongside other
479 apps, it's easy to do with Rack's `URLMap`:
480
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
481 ``` ruby
482 require 'resque/server'
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
483
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
484 run Rack::URLMap.new \
485 "/" => Your::App.new,
486 "/resque" => Resque::Server.new
487 ```
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
488
489 Check `examples/demo/config.ru` for a functional example (including
490 HTTP basic auth).
491
7976e93 @manuelmeurer Added comment on how to mount the Resque frontend in an existing Rail…
manuelmeurer authored
492 ### Rails 3
493
eb999e5 @federomero Add ' to "require resque/server" in README.markdown
federomero authored
494 You can also mount Resque on a subpath in your existing Rails 3 app by adding `require 'resque/server'` to the top of your routes file or in an initializer then adding this to `routes.rb`:
7976e93 @manuelmeurer Added comment on how to mount the Resque frontend in an existing Rail…
manuelmeurer authored
495
496 ``` ruby
497 mount Resque::Server.new, :at => "/resque"
498 ```
499
70d02b6 @defunkt explain how to setup the resque sinatra app
defunkt authored
500
e858376 @defunkt resque vs dj
defunkt authored
501 Resque vs DelayedJob
502 --------------------
503
504 How does Resque compare to DelayedJob, and why would you choose one
505 over the other?
506
507 * Resque supports multiple queues
508 * DelayedJob supports finer grained priorities
509 * Resque workers are resilient to memory leaks / bloat
510 * DelayedJob workers are extremely simple and easy to modify
511 * Resque requires Redis
512 * DelayedJob requires ActiveRecord
513 * Resque can only place JSONable Ruby objects on a queue as arguments
514 * DelayedJob can place _any_ Ruby object on its queue as arguments
515 * Resque includes a Sinatra app for monitoring what's going on
2be99f6 @jgeiger fix more spelling errors
jgeiger authored
516 * DelayedJob can be queried from within your Rails app if you want to
e858376 @defunkt resque vs dj
defunkt authored
517 add an interface
518
519 If you're doing Rails development, you already have a database and
520 ActiveRecord. DelayedJob is super easy to setup and works great.
521 GitHub used it for many months to process almost 200 million jobs.
522
523 Choose Resque if:
524
525 * You need multiple queues
526 * You don't care / dislike numeric priorities
a26c405 @defunkt tweakage
defunkt authored
527 * You don't need to persist every Ruby object ever
e858376 @defunkt resque vs dj
defunkt authored
528 * You have potentially huge queues
529 * You want to see what's going on
530 * You expect a lot of failure / chaos
531 * You can setup Redis
b7763f0 @defunkt resque eats ram
defunkt authored
532 * You're not running short on RAM
e858376 @defunkt resque vs dj
defunkt authored
533
534 Choose DelayedJob if:
535
536 * You like numeric priorities
537 * You're not doing a gigantic amount of jobs each day
538 * Your queue stays small and nimble
539 * There is not a lot failure / chaos
540 * You want to easily throw anything on the queue
541 * You don't want to setup Redis
542
543 In no way is Resque a "better" DelayedJob, so make sure you pick the
544 tool that's best for your app.
545
d974db2 @defunkt readme tweaks
defunkt authored
546
4ef73c7 @defunkt install redis
defunkt authored
547 Installing Redis
548 ----------------
d974db2 @defunkt readme tweaks
defunkt authored
549
4232321 @defunkt Mention Redis version
defunkt authored
550 Resque requires Redis 0.900 or higher.
551
5bcd1c0 @defunkt readme tweaks
defunkt authored
552 Resque uses Redis' lists for its queues. It also stores worker state
553 data in Redis.
554
4ef73c7 @defunkt install redis
defunkt authored
555 #### Homebrew
d974db2 @defunkt readme tweaks
defunkt authored
556
5bcd1c0 @defunkt readme tweaks
defunkt authored
557 If you're on OS X, Homebrew is the simplest way to install Redis:
d974db2 @defunkt readme tweaks
defunkt authored
558
559 $ brew install redis
560 $ redis-server /usr/local/etc/redis.conf
561
fdf1a87 @defunkt installing redis
defunkt authored
562 You now have a Redis daemon running on 6379.
563
4ef73c7 @defunkt install redis
defunkt authored
564 #### Via Resque
fdf1a87 @defunkt installing redis
defunkt authored
565
566 Resque includes Rake tasks (thanks to Ezra's redis-rb) that will
567 install and run Redis for you:
568
569 $ git clone git://github.com/defunkt/resque.git
570 $ cd resque
571 $ rake redis:install dtach:install
572 $ rake redis:start
573
69e1077 Modified rake tasks to allow for a lack of admin access. See README.m…
Rob Hanlon authored
574 Or, if you don't have admin access on your machine:
575
576 $ git clone git://github.com/defunkt/resque.git
577 $ cd resque
51e8841 Fixed README.markdown
Rob Hanlon authored
578 $ PREFIX=<your_prefix> rake redis:install dtach:install
69e1077 Modified rake tasks to allow for a lack of admin access. See README.m…
Rob Hanlon authored
579 $ rake redis:start
580
fdf1a87 @defunkt installing redis
defunkt authored
581 You now have Redis running on 6379. Wait a second then hit ctrl-\ to
582 detach and keep it running in the background.
583
4ef73c7 @defunkt install redis
defunkt authored
584 The demo is probably the best way to figure out how to put the parts
585 together. But, it's not that hard.
586
587
9976038 @defunkt dependencies
defunkt authored
588 Resque Dependencies
589 -------------------
590
7f4c79a @defunkt bundle install
defunkt authored
591 $ gem install bundler
592 $ bundle install
4232321 @defunkt Mention Redis version
defunkt authored
593
9976038 @defunkt dependencies
defunkt authored
594
5bcd1c0 @defunkt readme tweaks
defunkt authored
595 Installing Resque
596 -----------------
fdf1a87 @defunkt installing redis
defunkt authored
597
598 ### In a Rack app, as a gem
599
600 First install the gem.
601
3d9c45a @defunkt remove gemcutter noise
defunkt authored
602 $ gem install resque
fdf1a87 @defunkt installing redis
defunkt authored
603
604 Next include it in your application.
605
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
606 ``` ruby
607 require 'resque'
608 ```
fdf1a87 @defunkt installing redis
defunkt authored
609
610 Now start your application:
611
612 rackup config.ru
613
614 That's it! You can now create Resque jobs from within your app.
615
616 To start a worker, create a Rakefile in your app's root (or add this
617 to an existing Rakefile):
618
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
619 ``` ruby
620 require 'your/app'
621 require 'resque/tasks'
622 ```
d974db2 @defunkt readme tweaks
defunkt authored
623
fdf1a87 @defunkt installing redis
defunkt authored
624 Now:
625
626 $ QUEUE=* rake resque:work
f116bac @defunkt whitespace
defunkt authored
627
fdf1a87 @defunkt installing redis
defunkt authored
628 Alternately you can define a `resque:setup` hook in your Rakefile if you
629 don't want to load your app every time rake runs.
d974db2 @defunkt readme tweaks
defunkt authored
630
631
1f1ff49 @mrduncan Adding install instructions for Rails 3
mrduncan authored
632 ### In a Rails 2.x app, as a gem
d974db2 @defunkt readme tweaks
defunkt authored
633
fdf1a87 @defunkt installing redis
defunkt authored
634 First install the gem.
d974db2 @defunkt readme tweaks
defunkt authored
635
3d9c45a @defunkt remove gemcutter noise
defunkt authored
636 $ gem install resque
d974db2 @defunkt readme tweaks
defunkt authored
637
fdf1a87 @defunkt installing redis
defunkt authored
638 Next include it in your application.
d974db2 @defunkt readme tweaks
defunkt authored
639
fdf1a87 @defunkt installing redis
defunkt authored
640 $ cat config/initializers/load_resque.rb
f116bac @defunkt whitespace
defunkt authored
641 require 'resque'
d974db2 @defunkt readme tweaks
defunkt authored
642
fdf1a87 @defunkt installing redis
defunkt authored
643 Now start your application:
d974db2 @defunkt readme tweaks
defunkt authored
644
fdf1a87 @defunkt installing redis
defunkt authored
645 $ ./script/server
646
647 That's it! You can now create Resque jobs from within your app.
648
81a5cc2 @defunkt readme tweaks
defunkt authored
649 To start a worker, add this to your Rakefile in `RAILS_ROOT`:
fdf1a87 @defunkt installing redis
defunkt authored
650
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
651 ``` ruby
652 require 'resque/tasks'
653 ```
fdf1a87 @defunkt installing redis
defunkt authored
654
655 Now:
656
657 $ QUEUE=* rake environment resque:work
f116bac @defunkt whitespace
defunkt authored
658
fdf1a87 @defunkt installing redis
defunkt authored
659 Don't forget you can define a `resque:setup` hook in
660 `lib/tasks/whatever.rake` that loads the `environment` task every time.
661
662
1f1ff49 @mrduncan Adding install instructions for Rails 3
mrduncan authored
663 ### In a Rails 2.x app, as a plugin
fdf1a87 @defunkt installing redis
defunkt authored
664
665 $ ./script/plugin install git://github.com/defunkt/resque
666
667 That's it! Resque will automatically be available when your Rails app
668 loads.
669
670 To start a worker:
671
672 $ QUEUE=* rake environment resque:work
f116bac @defunkt whitespace
defunkt authored
673
fdf1a87 @defunkt installing redis
defunkt authored
674 Don't forget you can define a `resque:setup` hook in
675 `lib/tasks/whatever.rake` that loads the `environment` task every time.
d974db2 @defunkt readme tweaks
defunkt authored
676
677
1f1ff49 @mrduncan Adding install instructions for Rails 3
mrduncan authored
678 ### In a Rails 3 app, as a gem
679
680 First include it in your Gemfile.
681
682 $ cat Gemfile
683 ...
684 gem 'resque'
685 ...
686
687 Next install it with Bundler.
688
689 $ bundle install
690
691 Now start your application:
692
693 $ rails server
694
695 That's it! You can now create Resque jobs from within your app.
696
697 To start a worker, add this to a file in `lib/tasks` (ex:
698 `lib/tasks/resque.rake`):
699
700 ``` ruby
701 require 'resque/tasks'
702 ```
703
704 Now:
705
706 $ QUEUE=* rake environment resque:work
707
708 Don't forget you can define a `resque:setup` hook in
709 `lib/tasks/whatever.rake` that loads the `environment` task every time.
710
711
17c857a @defunkt document configuration
defunkt authored
712 Configuration
713 -------------
714
715 You may want to change the Redis host and port Resque connects to, or
716 set various other options at startup.
717
718 Resque has a `redis` setter which can be given a string or a Redis
719 object. This means if you're already using Redis in your app, Resque
720 can re-use the existing connection.
721
81a5cc2 @defunkt readme tweaks
defunkt authored
722 String: `Resque.redis = 'localhost:6379'`
17c857a @defunkt document configuration
defunkt authored
723
ee69b84 @defunkt mention namespace options in the readme
defunkt authored
724 Redis: `Resque.redis = $redis`
17c857a @defunkt document configuration
defunkt authored
725
726 For our rails app we have a `config/initializers/resque.rb` file where
727 we load `config/resque.yml` by hand and set the Redis information
728 appropriately.
729
730 Here's our `config/resque.yml`:
731
732 development: localhost:6379
733 test: localhost:6379
734 staging: redis1.se.github.com:6379
735 fi: localhost:6379
736 production: redis1.ae.github.com:6379
737
738 And our initializer:
739
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
740 ``` ruby
741 rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
742 rails_env = ENV['RAILS_ENV'] || 'development'
17c857a @defunkt document configuration
defunkt authored
743
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
744 resque_config = YAML.load_file(rails_root + '/config/resque.yml')
745 Resque.redis = resque_config[rails_env]
746 ```
17c857a @defunkt document configuration
defunkt authored
747
748 Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
749 this way we can tell our Sinatra app about the config file:
750
81a5cc2 @defunkt readme tweaks
defunkt authored
751 $ RAILS_ENV=production resque-web rails_root/config/initializers/resque.rb
17c857a @defunkt document configuration
defunkt authored
752
753 Now everyone is on the same page.
754
b4fcfa7 @bogdan Resque.inline configuration support
bogdan authored
755 Also, you could disable jobs queueing by setting 'inline' attribute.
756 For example, if you want to run all jobs in the same process for cucumber, try:
757
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
758 ``` ruby
759 Resque.inline = ENV['RAILS_ENV'] == "cucumber"
760 ```
b4fcfa7 @bogdan Resque.inline configuration support
bogdan authored
761
7158bd8 @defunkt Worker Hooks get their own section
defunkt authored
762
aebbefd @defunkt HOOKS.md
defunkt authored
763 Plugins and Hooks
764 -----------------
17c857a @defunkt document configuration
defunkt authored
765
124be1b @defunkt Worker Hooks goes into PLUGINS.md
defunkt authored
766 For a list of available plugins see
767 <http://wiki.github.com/defunkt/resque/plugins>.
a145eda @defunkt docs for hooks
defunkt authored
768
aebbefd @defunkt HOOKS.md
defunkt authored
769 If you'd like to write your own plugin, or want to customize Resque
770 using hooks (such as `Resque.after_fork`), see
e8af6a3 @defunkt Fix hooks link
defunkt authored
771 [docs/HOOKS.md](http://github.com/defunkt/resque/blob/master/docs/HOOKS.md).
a145eda @defunkt docs for hooks
defunkt authored
772
7158bd8 @defunkt Worker Hooks get their own section
defunkt authored
773
ee69b84 @defunkt mention namespace options in the readme
defunkt authored
774 Namespaces
775 ----------
776
777 If you're running multiple, separate instances of Resque you may want
778 to namespace the keyspaces so they do not overlap. This is not unlike
779 the approach taken by many memcached clients.
780
781 This feature is provided by the [redis-namespace][rs] library, which
782 Resque uses by default to separate the keys it manages from other keys
783 in your Redis server.
784
785 Simply use the `Resque.redis.namespace` accessor:
786
f56fa99 @defunkt Highlight Ruby in the README
defunkt authored
787 ``` ruby
788 Resque.redis.namespace = "resque:GitHub"
789 ```
ee69b84 @defunkt mention namespace options in the readme
defunkt authored
790
791 We recommend sticking this in your initializer somewhere after Redis
792 is configured.
793
794
5bcd1c0 @defunkt readme tweaks
defunkt authored
795 Demo
796 ----
797
798 Resque ships with a demo Sinatra app for creating jobs that are later
799 processed in the background.
800
801 Try it out by looking at the README, found at `examples/demo/README.markdown`.
d974db2 @defunkt readme tweaks
defunkt authored
802
803
d5c6d1b @defunkt mention example god configs in the README
defunkt authored
804 Monitoring
805 ----------
806
ccfa00b @defunkt mention monit
defunkt authored
807 ### god
808
d5c6d1b @defunkt mention example god configs in the README
defunkt authored
809 If you're using god to monitor Resque, we have provided example
810 configs in `examples/god/`. One is for starting / stopping workers,
811 the other is for killing workers that have been running too long.
812
ccfa00b @defunkt mention monit
defunkt authored
813 ### monit
814
815 If you're using monit, `examples/monit/resque.monit` is provided free
816 of charge. This is **not** used by GitHub in production, so please
817 send patches for any tweaks or improvements you can make to it.
818
d5c6d1b @defunkt mention example god configs in the README
defunkt authored
819
8d23a30 @defunkt FAQ
defunkt authored
820 Questions
821 ---------
822
823 Please add them to the [FAQ](https://github.com/defunkt/resque/wiki/FAQ) or
824 ask on the Mailing List. The Mailing List is explained further below
825
826
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
827 Development
828 -----------
829
830 Want to hack on Resque?
831
832 First clone the repo and run the tests:
833
834 git clone git://github.com/defunkt/resque.git
835 cd resque
836 rake test
837
838 If the tests do not pass make sure you have Redis installed
839 correctly (though we make an effort to tell you if we feel this is the
840 case). The tests attempt to start an isolated instance of Redis to
841 run against.
842
2be99f6 @jgeiger fix more spelling errors
jgeiger authored
843 Also make sure you've installed all the dependencies correctly. For
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
844 example, try loading the `redis-namespace` gem after you've installed
845 it:
846
847 $ irb
848 >> require 'rubygems'
849 => true
850 >> require 'redis/namespace'
851 => true
852
853 If you get an error requiring any of the dependencies, you may have
854 failed to install them or be seeing load path issues.
855
856 Feel free to ping the mailing list with your problem and we'll try to
857 sort it out.
858
859
860 Contributing
861 ------------
862
461351c @defunkt Reading the 'Contributing' wiki page is now required.
defunkt authored
863 Read the [Contributing][cb] wiki page first.
864
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
865 Once you've made your great commits:
866
7201f92 @jgeiger fix the links for forking and issues
jgeiger authored
867 1. [Fork][1] Resque
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
868 2. Create a topic branch - `git checkout -b my_branch`
869 3. Push to your branch - `git push origin my_branch`
f687521 @kneath Suggest sending pull requests instead of issues
kneath authored
870 4. Create a [Pull Request](http://help.github.com/pull-requests/) from your branch
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
871 5. That's it!
872
873
874 Mailing List
875 ------------
876
877 To join the list simply send an email to <resque@librelist.com>. This
878 will subscribe you and send you information about your subscription,
8ddefc7 @defunkt typo
defunkt authored
879 including unsubscribe information.
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
880
37385f1 @defunkt new archive url
defunkt authored
881 The archive can be found at <http://librelist.com/browser/resque/>.
3ecfa3f @defunkt add development / contributing / mailing list items
defunkt authored
882
883
9e48b3e @defunkt update the meta
defunkt authored
884 Meta
885 ----
886
887 * Code: `git clone git://github.com/defunkt/resque.git`
888 * Home: <http://github.com/defunkt/resque>
889698b @defunkt gh pages
defunkt authored
889 * Docs: <http://defunkt.github.com/resque/>
9e48b3e @defunkt update the meta
defunkt authored
890 * Bugs: <http://github.com/defunkt/resque/issues>
891 * List: <resque@librelist.com>
1ab1d57 @defunkt irc link
defunkt authored
892 * Chat: <irc://irc.freenode.net/resque>
9e48b3e @defunkt update the meta
defunkt authored
893 * Gems: <http://gemcutter.org/gems/resque>
94d5ff9 @defunkt readme
defunkt authored
894
988c4f0 @defunkt semantic versioning
defunkt authored
895 This project uses [Semantic Versioning][sv].
896
a26c405 @defunkt tweakage
defunkt authored
897
898 Author
899 ------
900
5bcd1c0 @defunkt readme tweaks
defunkt authored
901 Chris Wanstrath :: chris@ozmm.org :: @defunkt
c66b509 @defunkt link links
defunkt authored
902
7ec3141 @defunkt update blog url
defunkt authored
903 [0]: http://github.com/blog/542-introducing-resque
7201f92 @jgeiger fix the links for forking and issues
jgeiger authored
904 [1]: http://help.github.com/forking/
905 [2]: http://github.com/defunkt/resque/issues
988c4f0 @defunkt semantic versioning
defunkt authored
906 [sv]: http://semver.org/
ee69b84 @defunkt mention namespace options in the readme
defunkt authored
907 [rs]: http://github.com/defunkt/redis-namespace
7158bd8 @defunkt Worker Hooks get their own section
defunkt authored
908 [cb]: http://wiki.github.com/defunkt/resque/contributing
Something went wrong with that request. Please try again.