Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 211 lines (147 sloc) 6.752 kb
d0378cd @tobi Lets try this again as textile instead of markdown
tobi authored
1 h1. Delayed::Job
75b49dc @tobi Initial extraction
tobi authored
2
3 Delated_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
4
5 It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
6
7 * sending massive newsletters
8 * image resizing
9 * http downloads
10 * updating smart collections
11 * updating solr, our search server, after product changes
12 * batch imports
13 * spam checks
d58a26d @bkeepers Update the README
bkeepers authored
14
15 h2. Installation
16
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
17 To install as a gem, add the following to @config/environment.rb@:
18
19 <pre>
a533c41 @jqr Referencing the right gem.
jqr authored
20 config.gem 'delayed_job'
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
21 </pre>
22
23 Rake tasks are not automatically loaded from gems, so you'll need to add the following to your Rakefile:
24
25 <pre>
26 begin
27 require 'delayed/tasks'
28 rescue LoadError
29 STDERR.puts "Run `rake gems:install` to install delayed_job"
30 end
31 </pre>
32
33 To install as a plugin:
34
d58a26d @bkeepers Update the README
bkeepers authored
35 <pre>
36 script/plugin install git://github.com/collectiveidea/delayed_job.git
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
37 </pre>
38
7bec196 @jqr Instructing user to setup a backend as part of installation.
jqr authored
39 After delayed_job is installed, you will need to setup the backend.
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
40
fb46714 @bkeepers Documentation about the backends
bkeepers authored
41 h2. Backends
42
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
43 delayed_job supports multiple backends for storing the job queue. There are currently implementations for Active Record, MongoMapper, and DataMapper.
44
45 h3. Active Record
46
47 The default is Active Record, which requires a jobs table.
fb46714 @bkeepers Documentation about the backends
bkeepers authored
48
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
49 <pre>
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
50 $ script/generate delayed_job
51 $ rake db:migrate
d58a26d @bkeepers Update the README
bkeepers authored
52 </pre>
53
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
54 h3. MongoMapper
fb46714 @bkeepers Documentation about the backends
bkeepers authored
55
eecb1a8 @bkeepers Add note about MongoMapper.setup
bkeepers authored
56 You must use @MongoMapper.setup@ in the initializer:
57
fb46714 @bkeepers Documentation about the backends
bkeepers authored
58 <pre>
eecb1a8 @bkeepers Add note about MongoMapper.setup
bkeepers authored
59 config = YAML::load(File.read(Rails.root.join('config/mongo.yml')))
60 MongoMapper.setup(config, Rails.env)
61
4096c47 @zbelzer Updates to the backend test structure
zbelzer authored
62 Delayed::Worker.backend = :mongo_mapper
fb46714 @bkeepers Documentation about the backends
bkeepers authored
63 </pre>
64
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
65 h3. DataMapper
66
67 <pre>
68 # config/initializers/delayed_job.rb
69 Delayed::Worker.backend = :data_mapper
70 Delayed::Worker.backend.auto_upgrade!
71 </pre>
72
d58a26d @bkeepers Update the README
bkeepers authored
73 h2. Queuing Jobs
74
eb94694 @bkeepers Deprecate #send_later and #send_at in favor of new #delay method
bkeepers authored
75 Call @.delay.method(params)@ on any object and it will be processed in the background.
d58a26d @bkeepers Update the README
bkeepers authored
76
77 <pre>
78 # without delayed_job
79 Notifier.deliver_signup(@user)
80
81 # with delayed_job
eb94694 @bkeepers Deprecate #send_later and #send_at in favor of new #delay method
bkeepers authored
82 Notifier.delay.deliver_signup @user
d58a26d @bkeepers Update the README
bkeepers authored
83 </pre>
84
85 If a method should always be run in the background, you can call @#handle_asynchronously@ after the method declaration:
86
87 <pre>
88 class Device
89 def deliver
90 # long running method
91 end
92 handle_asynchronously :deliver
93 end
94
95 device = Device.new
96 device.deliver
97 </pre>
98
99 h2. Running Jobs
100
d98d6e4 @bkeepers README updates
bkeepers authored
101 @script/delayed_job@ can be used to manage a background process which will start working off jobs. Make sure you've run `script/generate delayed_job`.
d58a26d @bkeepers Update the README
bkeepers authored
102
103 <pre>
2efd95e @bkeepers Remove the -e/--environment option until we can come up with a workar…
bkeepers authored
104 $ RAILS_ENV=production script/delayed_job start
105 $ RAILS_ENV=production script/delayed_job stop
d58a26d @bkeepers Update the README
bkeepers authored
106
107 # Runs two workers in separate processes.
2efd95e @bkeepers Remove the -e/--environment option until we can come up with a workar…
bkeepers authored
108 $ RAILS_ENV=production script/delayed_job -n 2 start
109 $ RAILS_ENV=production script/delayed_job stop
d58a26d @bkeepers Update the README
bkeepers authored
110 </pre>
111
112 Workers can be running on any computer, as long as they have access to the database and their clock is in sync. Keep in mind that each worker will check the database at least every 5 seconds.
113
114 You can also invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task with @CTRL-C@.
115
116 h2. Custom Jobs
117
118 Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table. Job objects are serialized to yaml so that they can later be resurrected by the job runner.
119
120 <pre>
121 class NewsletterJob < Struct.new(:text, :emails)
122 def perform
123 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
124 end
125 end
126
127 Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
128 </pre>
129
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
130 You can also add an optional on_permanent_failure method which will run if the job has failed too many times to be retried:
131
132 <pre>
133 class ParanoidNewsletterJob < NewsletterJob
134 def perform
135 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
136 end
137
138 def on_permanent_failure
139 page_sysadmin_in_the_middle_of_the_night
140 end
141 end
142 </pre>
143
d58a26d @bkeepers Update the README
bkeepers authored
144 h2. Gory Details
145
75b49dc @tobi Initial extraction
tobi authored
146 The library evolves around a delayed_jobs table which looks as follows:
147
800755a @bkeepers Fix formatting in README. closes #46
bkeepers authored
148 <pre>
149 create_table :delayed_jobs, :force => true do |table|
150 table.integer :priority, :default => 0 # Allows some jobs to jump to the front of the queue
151 table.integer :attempts, :default => 0 # Provides for retries, but still fail eventually.
152 table.text :handler # YAML-encoded string of the object that will do work
153 table.text :last_error # reason for last failure (See Note below)
154 table.datetime :run_at # When to run. Could be Time.zone.now for immediately, or sometime in the future.
155 table.datetime :locked_at # Set when a client is working on this object
156 table.datetime :failed_at # Set when all retries have failed (actually, by default, the record is deleted instead)
157 table.string :locked_by # Who is working on this object (if locked)
158 table.timestamps
159 end
160 </pre>
ad27c3e @dmag expand readme to include more hints
dmag authored
161
162 On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.
163
81601e4 @bkeepers Fix references to config options in docs
bkeepers authored
164 The default Worker.max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
ad27c3e @dmag expand readme to include more hints
dmag authored
165 With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours.
166
81601e4 @bkeepers Fix references to config options in docs
bkeepers authored
167 The default Worker.max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
ad27c3e @dmag expand readme to include more hints
dmag authored
168 make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.
169
170 By default, it will delete failed jobs (and it always deletes successful jobs). If you want to keep failed jobs, set
7a5c8f4 @bkeepers Move #reschedule from Job to Worker. Refs #26
bkeepers authored
171 Delayed::Worker.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.
ad27c3e @dmag expand readme to include more hints
dmag authored
172
173 Here is an example of changing job parameters in Rails:
174
d58a26d @bkeepers Update the README
bkeepers authored
175 <pre>
176 # config/initializers/delayed_job_config.rb
7a5c8f4 @bkeepers Move #reschedule from Job to Worker. Refs #26
bkeepers authored
177 Delayed::Worker.destroy_failed_jobs = false
178 Delayed::Worker.sleep_delay = 60
179 Delayed::Worker.max_attempts = 3
180 Delayed::Worker.max_run_time = 5.minutes
d58a26d @bkeepers Update the README
bkeepers authored
181 </pre>
75b49dc @tobi Initial extraction
tobi authored
182
d58a26d @bkeepers Update the README
bkeepers authored
183 h3. Cleaning up
75b49dc @tobi Initial extraction
tobi authored
184
d58a26d @bkeepers Update the README
bkeepers authored
185 You can invoke @rake jobs:clear@ to delete all jobs in the queue.
450908d @jbarnette Refactored jobs:work, added jobs:clear.
jbarnette authored
186
94be113 @bkeepers Added note about new mailing list
bkeepers authored
187 h2. Mailing List
188
189 Join us on the mailing list at http://groups.google.com/group/delayed_job
190
d58a26d @bkeepers Update the README
bkeepers authored
191 h2. How to contribute
75b49dc @tobi Initial extraction
tobi authored
192
d58a26d @bkeepers Update the README
bkeepers authored
193 If you find what looks like a bug:
ad27c3e @dmag expand readme to include more hints
dmag authored
194
d58a26d @bkeepers Update the README
bkeepers authored
195 # Check the GitHub issue tracker to see if anyone else has had the same issue.
196 http://github.com/collectiveidea/delayed_job/issues/
197 # If you don't see anything, create an issue with information on how to reproduce it.
ad27c3e @dmag expand readme to include more hints
dmag authored
198
d58a26d @bkeepers Update the README
bkeepers authored
199 If you want to contribute an enhancement or a fix:
450908d @jbarnette Refactored jobs:work, added jobs:clear.
jbarnette authored
200
d58a26d @bkeepers Update the README
bkeepers authored
201 # Fork the project on github.
202 http://github.com/collectiveidea/delayed_job/
203 # Make your changes with tests.
204 # Commit the changes without making changes to the Rakefile, VERSION, or any other files that aren't related to your enhancement or fix
205 # Send a pull request.
f2ea93c @tobi Small changes in the way th egems are build
tobi authored
206
31e6d07 @bkeepers Remove outdated changelog
bkeepers authored
207 h3. Changelog
f2ea93c @tobi Small changes in the way th egems are build
tobi authored
208
31e6d07 @bkeepers Remove outdated changelog
bkeepers authored
209 See http://wiki.github.com/collectiveidea/delayed_job/changelog for a list of changes.
f2ea93c @tobi Small changes in the way th egems are build
tobi authored
210
Something went wrong with that request. Please try again.