Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 247 lines (174 sloc) 8.099 kB
d0378cd @tobi Lets try this again as textile instead of markdown
tobi authored
1 h1. Delayed::Job
75b49dc @tobi Initial extraction
tobi authored
2
cdae1d2 @brandondrew Fixed typo in README
brandondrew authored
3 Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
75b49dc @tobi Initial extraction
tobi authored
4
5 It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
6
7 * sending massive newsletters
8 * image resizing
9 * http downloads
10 * updating smart collections
11 * updating solr, our search server, after product changes
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
12 * batch imports
13 * spam checks
d58a26d @bkeepers Update the README
bkeepers authored
14
57cda43 @betamatt Fix link textile formatting
betamatt authored
15 "Follow us on Twitter":https://twitter.com/delayedjob to get updates and notices about new releases.
16
6c564b4 @betamatt Add twitter link
betamatt authored
17
d58a26d @bkeepers Update the README
bkeepers authored
18 h2. Installation
19
1a39108 @bkeepers update link to v2.0 branch in README
bkeepers authored
20 delayed_job 2.1 only supports Rails 3.0+. See the "2.0 branch":https://github.com/collectiveidea/delayed_job/tree/v2.0 for Rails 2.
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
21
3655aab @bkeepers update README
bkeepers authored
22 To install, add delayed_job to your @Gemfile@ and run `bundle install`:
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
23
d58a26d @bkeepers Update the README
bkeepers authored
24 <pre>
3ff9711 @bkeepers Update README
bkeepers authored
25 gem 'delayed_job'
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
26 </pre>
27
7bec196 @jqr Instructing user to setup a backend as part of installation.
jqr authored
28 After delayed_job is installed, you will need to setup the backend.
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
29
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
30 h3. Backends
fb46714 @bkeepers Documentation about the backends
bkeepers authored
31
63f88d5 @bkeepers Link to wiki for info on other backends
bkeepers authored
32 delayed_job supports multiple backends for storing the job queue. "See the wiki for other backends":http://wiki.github.com/collectiveidea/delayed_job/backends besides Active Record.
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
33
34 The default is Active Record, which requires a jobs table.
fb46714 @bkeepers Documentation about the backends
bkeepers authored
35
491a564 @bkeepers Move rake task to lib and update directions for installing delayed_jo…
bkeepers authored
36 <pre>
3ff9711 @bkeepers Update README
bkeepers authored
37 $ script/rails generate delayed_job
39bfbe8 @bkeepers Tell DataMapper folks to run auto_upgrade
bkeepers authored
38 $ rake db:migrate
d58a26d @bkeepers Update the README
bkeepers authored
39 </pre>
40
41 h2. Queuing Jobs
42
eb94694 @bkeepers Deprecate #send_later and #send_at in favor of new #delay method
bkeepers authored
43 Call @.delay.method(params)@ on any object and it will be processed in the background.
d58a26d @bkeepers Update the README
bkeepers authored
44
45 <pre>
46 # without delayed_job
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
47 @user.activate!(@device)
d58a26d @bkeepers Update the README
bkeepers authored
48
49 # with delayed_job
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
50 @user.delay.activate!(@device)
d58a26d @bkeepers Update the README
bkeepers authored
51 </pre>
52
53 If a method should always be run in the background, you can call @#handle_asynchronously@ after the method declaration:
54
55 <pre>
56 class Device
57 def deliver
58 # long running method
59 end
60 handle_asynchronously :deliver
61 end
62
63 device = Device.new
64 device.deliver
65 </pre>
66
4784f9d Added options to handle_asynchronously
David Genord II authored
67 handle_asynchronously can take as options anything you can pass to delay. In addition the values can be Proc objects allowing call time evaluation of the value. For some examples:
68
69 <pre>
70 class LongTasks
71 def send_mailer
72 # Some other code
73 end
74 handle_asynchronously :send_mailer, :priority => 20
75
76 def in_the_future
77 # Some other code
78 end
79 # 5.minutes.from_now will be evaluated when in_the_future is called
80 handle_asynchronously :in_the_future, :run_at => Proc.new { 5.minutes.from_now }
81
82 def self.when_to_run
83 2.hours.from_now
84 end
85
86 def call_a_class_method
87 # Some other code
88 end
89 handle_asynchronously :call_a_class_method, :run_at => Proc.new { when_to_run }
90
91 attr_reader :how_important
92
93 def call_an_instance_method
94 # Some other code
95 end
96 handle_asynchronously :call_an_instance_method, :priority => Proc.new {|i| i.how_important }
97 end
98 </pre>
99
a1b8b56 @bkeepers Add section in README specifically for Rails 3 mailers
bkeepers authored
100 h3. Rails 3 Mailers
101
102 Due to how mailers are implemented in Rails 3, we had to do a little work around to get delayed_job to work.
103
104 <pre>
105 # without delayed_job
106 Notifier.signup(@user).deliver
107
108 # with delayed_job
109 Notifier.delay.signup(@user)
110 </pre>
111
112 Remove the @.deliver@ method to make it work. It's not ideal, but it's the best we could do for now.
113
d58a26d @bkeepers Update the README
bkeepers authored
114 h2. Running Jobs
115
d98d6e4 @bkeepers README updates
bkeepers authored
116 @script/delayed_job@ can be used to manage a background process which will start working off jobs. Make sure you've run `script/generate delayed_job`.
d58a26d @bkeepers Update the README
bkeepers authored
117
118 <pre>
2efd95e @bkeepers Remove the -e/--environment option until we can come up with a workar…
bkeepers authored
119 $ RAILS_ENV=production script/delayed_job start
120 $ RAILS_ENV=production script/delayed_job stop
d58a26d @bkeepers Update the README
bkeepers authored
121
122 # Runs two workers in separate processes.
2efd95e @bkeepers Remove the -e/--environment option until we can come up with a workar…
bkeepers authored
123 $ RAILS_ENV=production script/delayed_job -n 2 start
124 $ RAILS_ENV=production script/delayed_job stop
d58a26d @bkeepers Update the README
bkeepers authored
125 </pre>
126
127 Workers can be running on any computer, as long as they have access to the database and their clock is in sync. Keep in mind that each worker will check the database at least every 5 seconds.
128
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
129 You can also invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task with @CTRL-C@.
d58a26d @bkeepers Update the README
bkeepers authored
130
131 h2. Custom Jobs
132
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
133 Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table. Job objects are serialized to yaml so that they can later be resurrected by the job runner.
d58a26d @bkeepers Update the README
bkeepers authored
134
135 <pre>
136 class NewsletterJob < Struct.new(:text, :emails)
137 def perform
138 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
139 end
140 end
141
d58a26d @bkeepers Update the README
bkeepers authored
142 Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
143 </pre>
144
129a5d5 @bkeepers Update docs on hooks
bkeepers authored
145 h2. Hooks
146
147 You can define hooks on your job that will be called at different stages in the process:
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
148
149 <pre>
150 class ParanoidNewsletterJob < NewsletterJob
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
151 def enqueue(job)
152 record_stat 'newsletter_job/enqueue'
153 end
154
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
155 def perform
156 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
129a5d5 @bkeepers Update docs on hooks
bkeepers authored
157 end
158
159 def before(job)
160 record_stat 'newsletter_job/start'
161 end
162
163 def after(job)
164 record_stat 'newsletter_job/after'
165 end
166
167 def success(job)
168 record_stat 'newsletter_job/success'
169 end
170
171 def error(job, exception)
172 notify_hoptoad(exception)
173 end
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
174
129a5d5 @bkeepers Update docs on hooks
bkeepers authored
175 def failure
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
176 page_sysadmin_in_the_middle_of_the_night
177 end
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
178 end
d2f14cd Added on_permanent_failure hook
Phil Darnowsky authored
179 </pre>
180
d58a26d @bkeepers Update the README
bkeepers authored
181 h2. Gory Details
182
3e8a196 @bkeepers Added #enqueue hook
bkeepers authored
183 The library evolves around a delayed_jobs table which looks as follows:
75b49dc @tobi Initial extraction
tobi authored
184
800755a @bkeepers Fix formatting in README. closes #46
bkeepers authored
185 <pre>
186 create_table :delayed_jobs, :force => true do |table|
187 table.integer :priority, :default => 0 # Allows some jobs to jump to the front of the queue
188 table.integer :attempts, :default => 0 # Provides for retries, but still fail eventually.
189 table.text :handler # YAML-encoded string of the object that will do work
190 table.text :last_error # reason for last failure (See Note below)
191 table.datetime :run_at # When to run. Could be Time.zone.now for immediately, or sometime in the future.
192 table.datetime :locked_at # Set when a client is working on this object
193 table.datetime :failed_at # Set when all retries have failed (actually, by default, the record is deleted instead)
194 table.string :locked_by # Who is working on this object (if locked)
195 table.timestamps
196 end
197 </pre>
ad27c3e @dmag expand readme to include more hints
dmag authored
198
199 On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.
200
81601e4 @bkeepers Fix references to config options in docs
bkeepers authored
201 The default Worker.max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
ad27c3e @dmag expand readme to include more hints
dmag authored
202 With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours.
203
81601e4 @bkeepers Fix references to config options in docs
bkeepers authored
204 The default Worker.max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
ad27c3e @dmag expand readme to include more hints
dmag authored
205 make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.
206
207 By default, it will delete failed jobs (and it always deletes successful jobs). If you want to keep failed jobs, set
7a5c8f4 @bkeepers Move #reschedule from Job to Worker. Refs #26
bkeepers authored
208 Delayed::Worker.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.
ad27c3e @dmag expand readme to include more hints
dmag authored
209
7a4ed0d @eugenebolshakov Updated readme
eugenebolshakov authored
210 By default all jobs are scheduled with priority = 0, which is top priority. You can change this by setting Delayed::Worker.default_priority to something else. Lower numbers have higher priority.
211
b8a0cc5 @edwinv Added Delay::Worker.delay_jobs configuration to allow delayed job to …
edwinv authored
212 It is possible to disable delayed jobs for testing purposes. Set Delayed::Worker.delay_jobs = false to execute all jobs realtime.
213
ad27c3e @dmag expand readme to include more hints
dmag authored
214 Here is an example of changing job parameters in Rails:
215
d58a26d @bkeepers Update the README
bkeepers authored
216 <pre>
217 # config/initializers/delayed_job_config.rb
7a5c8f4 @bkeepers Move #reschedule from Job to Worker. Refs #26
bkeepers authored
218 Delayed::Worker.destroy_failed_jobs = false
219 Delayed::Worker.sleep_delay = 60
220 Delayed::Worker.max_attempts = 3
221 Delayed::Worker.max_run_time = 5.minutes
b8a0cc5 @edwinv Added Delay::Worker.delay_jobs configuration to allow delayed job to …
edwinv authored
222 Delayed::Worker.delay_jobs = !Rails.env.test?
d58a26d @bkeepers Update the README
bkeepers authored
223 </pre>
75b49dc @tobi Initial extraction
tobi authored
224
d58a26d @bkeepers Update the README
bkeepers authored
225 h3. Cleaning up
75b49dc @tobi Initial extraction
tobi authored
226
d58a26d @bkeepers Update the README
bkeepers authored
227 You can invoke @rake jobs:clear@ to delete all jobs in the queue.
450908d @jbarnette Refactored jobs:work, added jobs:clear.
jbarnette authored
228
94be113 @bkeepers Added note about new mailing list
bkeepers authored
229 h2. Mailing List
230
231 Join us on the mailing list at http://groups.google.com/group/delayed_job
232
d58a26d @bkeepers Update the README
bkeepers authored
233 h2. How to contribute
75b49dc @tobi Initial extraction
tobi authored
234
d58a26d @bkeepers Update the README
bkeepers authored
235 If you find what looks like a bug:
ad27c3e @dmag expand readme to include more hints
dmag authored
236
3ff9711 @bkeepers Update README
bkeepers authored
237 # Search the "mailing list":http://groups.google.com/group/delayed_job to see if anyone else had the same issue.
238 # Check the "GitHub issue tracker":http://github.com/collectiveidea/delayed_job/issues/ to see if anyone else has reported issue.
d58a26d @bkeepers Update the README
bkeepers authored
239 # If you don't see anything, create an issue with information on how to reproduce it.
ad27c3e @dmag expand readme to include more hints
dmag authored
240
d58a26d @bkeepers Update the README
bkeepers authored
241 If you want to contribute an enhancement or a fix:
450908d @jbarnette Refactored jobs:work, added jobs:clear.
jbarnette authored
242
d58a26d @bkeepers Update the README
bkeepers authored
243 # Fork the project on github.
244 # Make your changes with tests.
3ff9711 @bkeepers Update README
bkeepers authored
245 # Commit the changes without making changes to the Rakefile or any other files that aren't related to your enhancement or fix
d58a26d @bkeepers Update the README
bkeepers authored
246 # Send a pull request.
Something went wrong with that request. Please try again.