Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 105 lines (72 sloc) 3.857 kB
d0378cd @tobi Lets try this again as textile instead of markdown
authored
1 h1. Delayed::Job
75b49dc @tobi Initial extraction
authored
2
3 Delated_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
4
5 It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
6
7 * sending massive newsletters
8 * image resizing
9 * http downloads
10 * updating smart collections
11 * updating solr, our search server, after product changes
12 * batch imports
13 * spam checks
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
14
d0378cd @tobi Lets try this again as textile instead of markdown
authored
15 h2. Changes
5998931 Renamed locked_until to locked_at. We now store when we start a given…
Tobias Lütke authored
16
17 * 1.6 Renamed locked_until to locked_at. We now store when we start a given task instead of how long it will be locked by the worker. This allows us to get a reading on how long a task took to execute.
d0378cd @tobi Lets try this again as textile instead of markdown
authored
18 * 1.5 Job runners can now be run in parallel. Two new database columns are needed: locked_until and locked_by. This allows us to use pessimistic locking, which enables us to run as many worker processes as we need to speed up queue processing.
19 * 1.0 Initial release
75b49dc @tobi Initial extraction
authored
20
d0378cd @tobi Lets try this again as textile instead of markdown
authored
21 h2. Setup
75b49dc @tobi Initial extraction
authored
22
23 The library evolves around a delayed_jobs table which looks as follows:
24
25 create_table :delayed_jobs, :force => true do |table|
7931ef1 @technoweenie update delayed job schema in the readme
technoweenie authored
26 table.integer :priority, :default => 0
27 table.integer :attempts, :default => 0
28 table.text :handler
29 table.string :last_error
30 table.datetime :run_at
31 table.datetime :locked_at
a137def @defunkt Changes extracted from GitHub:
defunkt authored
32 table.datetime :failed_at
7931ef1 @technoweenie update delayed job schema in the readme
technoweenie authored
33 table.string :locked_by
34 table.timestamps
ce63517 @jbarnette Update the README's create_table to match the schema.
jbarnette authored
35 end
75b49dc @tobi Initial extraction
authored
36
d0378cd @tobi Lets try this again as textile instead of markdown
authored
37 h2. Usage
75b49dc @tobi Initial extraction
authored
38
39 Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table.
40 Job objects are serialized to yaml so that they can later be resurrected by the job runner.
41
42 class NewsletterJob < Struct.new(:text, :emails)
43 def perform
44 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
45 end
46 end
47
48 Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
49
50 There is also a second way to get jobs in the queue: send_later.
51
52
53 BatchImporter.new(Shop.find(1)).send_later(:import_massive_csv, massive_csv)
54
55
56 This will simply create a Delayed::PerformableMethod job in the jobs table which serializes all the parameters you pass to it. There are some special smarts for active record objects
57 which are stored as their text representation and loaded from the database fresh when the job is actually run later.
58
59
d0378cd @tobi Lets try this again as textile instead of markdown
authored
60 h2. Running the tasks
75b49dc @tobi Initial extraction
authored
61
0880b0f @tobi More formatting updates for readme
authored
62 You can invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task by @CTRL-C@.
75b49dc @tobi Initial extraction
authored
63
64 At Shopify we run the the tasks from a simple script/job_runner which is being invoked by runnit:
0880b0f @tobi More formatting updates for readme
authored
65
66 <pre><code>
75b49dc @tobi Initial extraction
authored
67 #!/usr/bin/env ruby
68 require File.dirname(__FILE__) + '/../config/environment'
0880b0f @tobi More formatting updates for readme
authored
69
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
70 SLEEP = 5
0880b0f @tobi More formatting updates for readme
authored
71
75b49dc @tobi Initial extraction
authored
72 trap('TERM') { puts 'Exiting...'; $exit = true }
73 trap('INT') { puts 'Exiting...'; $exit = true }
0880b0f @tobi More formatting updates for readme
authored
74
c689bf4 Fix spelling mistake
Tobias Lütke authored
75 puts "*** Starting job worker #{Delayed::Job.worker_name}"
0880b0f @tobi More formatting updates for readme
authored
76
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
77 begin
0880b0f @tobi More formatting updates for readme
authored
78
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
79 loop do
80 result = nil
75b49dc @tobi Initial extraction
authored
81
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
82 realtime = Benchmark.realtime do
83 result = Delayed::Job.work_off
84 end
75b49dc @tobi Initial extraction
authored
85
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
86 count = result.sum
87
88 break if $exit
75b49dc @tobi Initial extraction
authored
89
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
90 if count.zero?
91 sleep(SLEEP)
92 puts 'Waiting for more jobs...'
93 else
94 status = "#{count} jobs processed at %.4f j/s, %d failed ..." % [count / realtime, result.last]
95 RAILS_DEFAULT_LOGGER.info status
96 puts status
97 end
75b49dc @tobi Initial extraction
authored
98
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
99 break if $exit
75b49dc @tobi Initial extraction
authored
100 end
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
101 ensure
102 Delayed::Job.clear_locks!
103 end
0880b0f @tobi More formatting updates for readme
authored
104 </code></pre>
Something went wrong with that request. Please try again.