Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 103 lines (70 sloc) 3.735 kB
d0378cd @tobi Lets try this again as textile instead of markdown
authored
1 h1. Delayed::Job
75b49dc @tobi Initial extraction
authored
2
3 Delated_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
4
5 It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
6
7 * sending massive newsletters
8 * image resizing
9 * http downloads
10 * updating smart collections
11 * updating solr, our search server, after product changes
12 * batch imports
13 * spam checks
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
14
d0378cd @tobi Lets try this again as textile instead of markdown
authored
15 h2. Changes
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
16
d0378cd @tobi Lets try this again as textile instead of markdown
authored
17 * 1.5 Job runners can now be run in parallel. Two new database columns are needed: locked_until and locked_by. This allows us to use pessimistic locking, which enables us to run as many worker processes as we need to speed up queue processing.
18 * 1.0 Initial release
75b49dc @tobi Initial extraction
authored
19
d0378cd @tobi Lets try this again as textile instead of markdown
authored
20 h2. Setup
75b49dc @tobi Initial extraction
authored
21
22 The library evolves around a delayed_jobs table which looks as follows:
23
24 create_table :delayed_jobs, :force => true do |table|
25 table.integer :priority, :default => 0
26 table.integer :attempts, :default => 0
27 table.text :handler
28 table.string :last_error
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
29 table.datetime :run_at
30 table.datetime :locked_until
31 table.string :locked_by
75b49dc @tobi Initial extraction
authored
32 table.timestamps
33 end
34
d0378cd @tobi Lets try this again as textile instead of markdown
authored
35 h2. Usage
75b49dc @tobi Initial extraction
authored
36
37 Jobs are simple ruby objects with a method called perform. Any object which responds to perform can be stuffed into the jobs table.
38 Job objects are serialized to yaml so that they can later be resurrected by the job runner.
39
40 class NewsletterJob < Struct.new(:text, :emails)
41 def perform
42 emails.each { |e| NewsletterMailer.deliver_text_to_email(text, e) }
43 end
44 end
45
46 Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', Customers.find(:all).collect(&:email))
47
48 There is also a second way to get jobs in the queue: send_later.
49
50
51 BatchImporter.new(Shop.find(1)).send_later(:import_massive_csv, massive_csv)
52
53
54 This will simply create a Delayed::PerformableMethod job in the jobs table which serializes all the parameters you pass to it. There are some special smarts for active record objects
55 which are stored as their text representation and loaded from the database fresh when the job is actually run later.
56
57
d0378cd @tobi Lets try this again as textile instead of markdown
authored
58 h2. Running the tasks
75b49dc @tobi Initial extraction
authored
59
0880b0f @tobi More formatting updates for readme
authored
60 You can invoke @rake jobs:work@ which will start working off jobs. You can cancel the rake task by @CTRL-C@.
75b49dc @tobi Initial extraction
authored
61
62 At Shopify we run the the tasks from a simple script/job_runner which is being invoked by runnit:
0880b0f @tobi More formatting updates for readme
authored
63
64 <pre><code>
75b49dc @tobi Initial extraction
authored
65 #!/usr/bin/env ruby
66 require File.dirname(__FILE__) + '/../config/environment'
0880b0f @tobi More formatting updates for readme
authored
67
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
68 SLEEP = 5
0880b0f @tobi More formatting updates for readme
authored
69
75b49dc @tobi Initial extraction
authored
70 trap('TERM') { puts 'Exiting...'; $exit = true }
71 trap('INT') { puts 'Exiting...'; $exit = true }
0880b0f @tobi More formatting updates for readme
authored
72
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
73 puts "*** Staring job worker #{Delayed::Job.worker_name}"
0880b0f @tobi More formatting updates for readme
authored
74
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
75 begin
0880b0f @tobi More formatting updates for readme
authored
76
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
77 loop do
78 result = nil
75b49dc @tobi Initial extraction
authored
79
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
80 realtime = Benchmark.realtime do
81 result = Delayed::Job.work_off
82 end
75b49dc @tobi Initial extraction
authored
83
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
84 count = result.sum
85
86 break if $exit
75b49dc @tobi Initial extraction
authored
87
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
88 if count.zero?
89 sleep(SLEEP)
90 puts 'Waiting for more jobs...'
91 else
92 status = "#{count} jobs processed at %.4f j/s, %d failed ..." % [count / realtime, result.last]
93 RAILS_DEFAULT_LOGGER.info status
94 puts status
95 end
75b49dc @tobi Initial extraction
authored
96
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
97 break if $exit
75b49dc @tobi Initial extraction
authored
98 end
8ec934e @tobi Removed the global lock. Jobs can now processed in parallel by runnin…
authored
99 ensure
100 Delayed::Job.clear_locks!
101 end
0880b0f @tobi More formatting updates for readme
authored
102 </code></pre>
Something went wrong with that request. Please try again.