Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 244 lines (173 sloc) 7.996 kB
d0378cd @tobi Lets try this again as textile instead of markdown
tobi authored
1 h1. Delayed::Job
75b49dc @tobi Initial extraction
tobi authored
2
cdae1d2 @brandondrew Fixed typo in README
brandondrew authored
3 Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
75b49dc @tobi Initial extraction
tobi authored
4
5 It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
6
7 * sending massive newsletters
8 * image resizing
9 * http downloads
10 * updating smart collections
11 * updating solr, our search server, after product changes
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
12 * batch imports
13 * spam checks
d58a26d @bkeepers Update the README
bkeepers authored
14
15 h2. Installation
16
1a39108 @bkeepers update link to v2.0 branch in README
bkeepers authored
17 delayed_job 2.1 only supports Rails 3.0+. See the "2.0 branch":https://github.com/collectiveidea/delayed_job/tree/v2.0 for Rails 2.
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
18
3655aab @bkeepers update README
bkeepers authored
19 To install, add delayed_job to your @Gemfile@ and run `bundle install`:
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
20
d58a26d @bkeepers Update the README
bkeepers authored
21 <pre>
3ff9711 @bkeepers Update README
bkeepers authored
22 gem 'delayed_job'
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
23 </pre>
24
7bec196 @jqr Instructing user to setup a backend as part of installation.
jqr authored
25 After delayed_job is installed, you will need to setup the backend.
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
26
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
27 h3. Backends
fb46714 @bkeepers Documentation about the backends
bkeepers authored
28
63f88d5 @bkeepers Link to wiki for info on other backends
bkeepers authored
29 delayed_job supports multiple backends for storing the job queue. "See the wiki for other backends":http://wiki.github.com/collectiveidea/delayed_job/backends besides Active Record.
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
30
31 The default is Active Record, which requires a jobs table.
fb46714 @bkeepers Documentation about the backends
bkeepers authored
32
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
33 <pre>
3ff9711 @bkeepers Update README
bkeepers authored
34 $ script/rails generate delayed_job
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
35 $ rake db:migrate
d58a26d @bkeepers Update the README
bkeepers authored
36 </pre>
37
38 h2. Queuing Jobs
39
eb94694 @bkeepers Deprecate #send_later and #send_at in favor of new #delay method
bkeepers authored
40 Call @.delay.method(params)@ on any object and it will be processed in the background.
d58a26d @bkeepers Update the README
bkeepers authored
41
42 <pre>
43 # without delayed_job
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
44 @user.activate!(@device)
d58a26d @bkeepers Update the README
bkeepers authored
45
46 # with delayed_job
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
47 @user.delay.activate!(@device)
d58a26d @bkeepers Update the README
bkeepers authored
48 </pre>
49
50 If a method should always be run in the background, you can call @#handle_asynchronously@ after the method declaration:
51
52 <pre>
53 class Device
54 def deliver
55 # long running method
56 end
57 handle_asynchronously :deliver
58 end
59
60 device = Device.new
61 device.deliver
62 </pre>
63
4784f9d Added options to handle_asynchronously
David Genord II authored
64 handle_asynchronously can take as options anything you can pass to delay. In addition the values can be Proc objects allowing call time evaluation of the value. For some examples:
65
66 <pre>
67 class LongTasks
68 def send_mailer
69 # Some other code
70 end
71 handle_asynchronously :send_mailer, :priority => 20
72
73 def in_the_future
74 # Some other code
75 end
76 # 5.minutes.from_now will be evaluated when in_the_future is called
77 handle_asynchronously :in_the_future, :run_at => Proc.new { 5.minutes.from_now }
78
79 def self.when_to_run
80 2.hours.from_now
81 end
82
83 def call_a_class_method
84 # Some other code
85 end
86 handle_asynchronously :call_a_class_method, :run_at => Proc.new { when_to_run }
87
88 attr_reader :how_important
89
90 def call_an_instance_method
91 # Some other code
92 end
93 handle_asynchronously :call_an_instance_method, :priority => Proc.new {|i| i.how_important }
94 end
95 </pre>
96
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
97 h3. Rails 3 Mailers
98
99 Due to how mailers are implemented in Rails 3, we had to do a little work around to get delayed_job to work.
100
101 <pre>
102 # without delayed_job
103 Notifier.signup(@user).deliver
104
105 # with delayed_job
106 Notifier.delay.signup(@user)
107 </pre>
108
109 Remove the @.deliver@ method to make it work. It's not ideal, but it's the best we could do for now.
110
d58a26d @bkeepers Update the README
bkeepers authored
111 h2. Running Jobs
112
d98d6e4 @bkeepers README updates
bkeepers authored
113 @script/delayed_job@ can be used to manage a background process which will start working off jobs. Make sure you've run `script/generate delayed_job`.
d58a26d @bkeepers Update the README
bkeepers authored
114
115 <pre>
2efd95e @bkeepers Remove the -e/--environment option until we can come up with a workar…
bkeepers authored
116 $ RAILS_ENV=production script/delayed_job start
117 $ RAILS_ENV=production script/delayed_job stop
d58a26d @bkeepers Update the README
bkeepers authored
118
119 # Runs two workers in separate processes.
2efd95e @bkeepers Remove the -e/--environment option until we can come up with a workar…
bkeepers authored
120 $ RAILS_ENV=production script/delayed_job -n 2 start
121 $ RAILS_ENV=production script/delayed_job stop
d58a26d @bkeepers Update the README
bkeepers authored
122 </pre>
123
124 Workers can be running on any computer, as long as they have access to the database and their clock is in sync. Keep in mind that each worker will check the database at least every 5 seconds.
125
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
126 You can also invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task with @CTRL-C@.
d58a26d @bkeepers Update the README
bkeepers authored
127
128 h2. Custom Jobs
129
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
130 Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table. Job objects are serialized to yaml so that they can later be resurrected by the job runner.
d58a26d @bkeepers Update the README
bkeepers authored
131
132 <pre>
133 class NewsletterJob < Struct.new(:text, :emails)
134 def perform
135 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
136 end
137 end
138
d58a26d @bkeepers Update the README
bkeepers authored
139 Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
140 </pre>
141
129a5d5 @bkeepers Update docs on hooks
bkeepers authored
142 h2. Hooks
143
144 You can define hooks on your job that will be called at different stages in the process:
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
145
146 <pre>
147 class ParanoidNewsletterJob < NewsletterJob
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
148 def enqueue(job)
149 record_stat 'newsletter_job/enqueue'
150 end
151
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
152 def perform
153 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
129a5d5 @bkeepers Update docs on hooks
bkeepers authored
154 end
155
156 def before(job)
157 record_stat 'newsletter_job/start'
158 end
159
160 def after(job)
161 record_stat 'newsletter_job/after'
162 end
163
164 def success(job)
165 record_stat 'newsletter_job/success'
166 end
167
168 def error(job, exception)
169 notify_hoptoad(exception)
170 end
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
171
129a5d5 @bkeepers Update docs on hooks
bkeepers authored
172 def failure
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
173 page_sysadmin_in_the_middle_of_the_night
174 end
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
175 end
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
176 </pre>
177
d58a26d @bkeepers Update the README
bkeepers authored
178 h2. Gory Details
179
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
180 The library evolves around a delayed_jobs table which looks as follows:
75b49dc @tobi Initial extraction
tobi authored
181
800755a @bkeepers Fix formatting in README. closes #46
bkeepers authored
182 <pre>
183 create_table :delayed_jobs, :force => true do |table|
184 table.integer :priority, :default => 0 # Allows some jobs to jump to the front of the queue
185 table.integer :attempts, :default => 0 # Provides for retries, but still fail eventually.
186 table.text :handler # YAML-encoded string of the object that will do work
187 table.text :last_error # reason for last failure (See Note below)
188 table.datetime :run_at # When to run. Could be Time.zone.now for immediately, or sometime in the future.
189 table.datetime :locked_at # Set when a client is working on this object
190 table.datetime :failed_at # Set when all retries have failed (actually, by default, the record is deleted instead)
191 table.string :locked_by # Who is working on this object (if locked)
192 table.timestamps
193 end
194 </pre>
ad27c3e @dmag expand readme to include more hints
dmag authored
195
196 On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.
197
81601e4 @bkeepers Fix references to config options in docs
bkeepers authored
198 The default Worker.max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
ad27c3e @dmag expand readme to include more hints
dmag authored
199 With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours.
200
81601e4 @bkeepers Fix references to config options in docs
bkeepers authored
201 The default Worker.max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
ad27c3e @dmag expand readme to include more hints
dmag authored
202 make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.
203
204 By default, it will delete failed jobs (and it always deletes successful jobs). If you want to keep failed jobs, set
7a5c8f4 @bkeepers Move #reschedule from Job to Worker. Refs #26
bkeepers authored
205 Delayed::Worker.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.
ad27c3e @dmag expand readme to include more hints
dmag authored
206
7a4ed0d @eugenebolshakov Updated readme
eugenebolshakov authored
207 By default all jobs are scheduled with priority = 0, which is top priority. You can change this by setting Delayed::Worker.default_priority to something else. Lower numbers have higher priority.
208
b8a0cc5 @edwinv Added Delay::Worker.delay_jobs configuration to allow delayed job to …
edwinv authored
209 It is possible to disable delayed jobs for testing purposes. Set Delayed::Worker.delay_jobs = false to execute all jobs realtime.
210
ad27c3e @dmag expand readme to include more hints
dmag authored
211 Here is an example of changing job parameters in Rails:
212
d58a26d @bkeepers Update the README
bkeepers authored
213 <pre>
214 # config/initializers/delayed_job_config.rb
7a5c8f4 @bkeepers Move #reschedule from Job to Worker. Refs #26
bkeepers authored
215 Delayed::Worker.destroy_failed_jobs = false
216 Delayed::Worker.sleep_delay = 60
217 Delayed::Worker.max_attempts = 3
218 Delayed::Worker.max_run_time = 5.minutes
b8a0cc5 @edwinv Added Delay::Worker.delay_jobs configuration to allow delayed job to …
edwinv authored
219 Delayed::Worker.delay_jobs = !Rails.env.test?
d58a26d @bkeepers Update the README
bkeepers authored
220 </pre>
75b49dc @tobi Initial extraction
tobi authored
221
d58a26d @bkeepers Update the README
bkeepers authored
222 h3. Cleaning up
75b49dc @tobi Initial extraction
tobi authored
223
d58a26d @bkeepers Update the README
bkeepers authored
224 You can invoke @rake jobs:clear@ to delete all jobs in the queue.
450908d @jbarnette Refactored jobs:work, added jobs:clear.
jbarnette authored
225
94be113 @bkeepers Added note about new mailing list
bkeepers authored
226 h2. Mailing List
227
228 Join us on the mailing list at http://groups.google.com/group/delayed_job
229
d58a26d @bkeepers Update the README
bkeepers authored
230 h2. How to contribute
75b49dc @tobi Initial extraction
tobi authored
231
d58a26d @bkeepers Update the README
bkeepers authored
232 If you find what looks like a bug:
ad27c3e @dmag expand readme to include more hints
dmag authored
233
3ff9711 @bkeepers Update README
bkeepers authored
234 # Search the "mailing list":http://groups.google.com/group/delayed_job to see if anyone else had the same issue.
235 # Check the "GitHub issue tracker":http://github.com/collectiveidea/delayed_job/issues/ to see if anyone else has reported issue.
d58a26d @bkeepers Update the README
bkeepers authored
236 # If you don't see anything, create an issue with information on how to reproduce it.
ad27c3e @dmag expand readme to include more hints
dmag authored
237
d58a26d @bkeepers Update the README
bkeepers authored
238 If you want to contribute an enhancement or a fix:
450908d @jbarnette Refactored jobs:work, added jobs:clear.
jbarnette authored
239
d58a26d @bkeepers Update the README
bkeepers authored
240 # Fork the project on github.
241 # Make your changes with tests.
3ff9711 @bkeepers Update README
bkeepers authored
242 # Commit the changes without making changes to the Rakefile or any other files that aren't related to your enhancement or fix
d58a26d @bkeepers Update the README
bkeepers authored
243 # Send a pull request.
Something went wrong with that request. Please try again.