Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 243 lines (207 sloc) 10.318 kb
2dba10f @mpasternacki Initial import
authored
1 # Dynamic Resque Pool
2
3 [Resque pool](https://github.com/nevans/resque-pool) by Nicholas Evans
4 is a library for managing a pool of
5 [resque](http://github.com/defunkt/resque) workers. Given a a config
6 file, it manages your workers for you, starting up the appropriate
7 number of workers for each worker type.
8
9 While the resque pool is convenient for a permanent process
10 overlooking a fairly constant set of workers, it is less than
11 convenient to use for more dynamic tasks - one-time long running batch
12 processing worker families that need to be started and supervised
13 manually, and often require adjusting number of worker processes for
14 maximum performance.
15
16 Workflow for this kind of task would be:
17
18 * prepare a yaml file with number of workers
19 * start resque-pool
20 * `tail -f` the log file on the other console
21 * look into `pstree` (or `ps faxuw` on Linux)
22 * babysit the process, check the system load whether number of workers
23 needs to be adjusted
24 * edit the yaml file
25 * `kill -HUP` the resque pool master (hope you remembered the PID from
26 pstree!)
27 * rinse, repeat
28
29 Or you can add `gem "resque-pool-dynamic"` to your Gemfile, `require
30 'resque/pool/dynamic/tasks'` to your Rakefile, and start an
31 interactive CLI interface that controls the resque-pool-master.
32
33 ## Example resque-pool-manager session
34
35 $ bundle exec rake resque:pool:dynamic WORKERS=foo=2:bar=2:\*=1
36 resque-pool-manager[21134]: started manager
37
38 Status: running, pid: 21134
39 Configuration:
40 bar: 2
41 "*": 1
42 foo: 2
43 Process tree:
44 -+= 21134 japhy resque-pool-master: managing [21136, 21135, 21137, 21138, 21141]
45 |--- 21135 japhy resque-1.20.0: Waiting for bar
46 |--- 21136 japhy resque-1.20.0: Waiting for bar
47 |--- 21137 japhy resque-1.20.0: Waiting for *
48 |--- 21138 japhy resque-1.20.0: Waiting for foo
49 \--- 21141 japhy resque-1.20.0: Waiting for foo
50
51 >> log.tail
52 resque-pool-manager[21134]: Pool contains worker PIDs: [21136, 21135, 21137, 21138, 21141]
53 resque-pool-worker[21135]: Starting worker portinari.local:21135:bar
54 resque-pool-worker[21137]: Starting worker portinari.local:21137:*
55
56 resque-pool-worker[21136]: Starting worker portinari.local:21136:bar
57 resque-pool-worker[21141]: Starting worker portinari.local:21141:foo
58 resque-pool-worker[21138]: Starting worker portinari.local:21138:foo
59
60 (log.tail is following the logfile like `tail -f` until you break with ^C)
61
62 ^C>> config :foo => 1, :bar => 4
63 Reloading resque-pool-master 21134 configuration
64 => {"bar"=>4, "*"=>1, "foo"=>1}
65 >> status
66
67 Status: running, pid: 21134
68 Configuration:
69 bar: 4
70 "*": 1
71 foo: 1
72 Process tree:
73 -+= 21134 japhy resque-pool-master: managing [21176, 21178, 21162, 21163, 21180, 21181]
74 |--- 21162 japhy resque-1.20.0: Waiting for bar
75 |--- 21163 japhy resque-1.20.0: Waiting for bar
76 |--- 21176 japhy resque-1.20.0: Waiting for bar
77 |--- 21178 japhy resque-1.20.0: Waiting for bar
78 |--- 21180 japhy resque-1.20.0: Waiting for *
79 \--- 21181 japhy resque-1.20.0: Waiting for foo
80
81 >> log.tail
82 resque-pool-manager[21134]: HUP: reload config file and reload logfiles
83 resque-pool-manager[21134]: Flushing logs
84 resque-pool-manager[21134]: HUP: gracefully shutdown old children (which have old logfiles open)
85 resque-pool-manager[21134]: HUP: new children will inherit new logfiles
86 resque-pool-worker[21163]: Starting worker portinari.local:21163:bar
87 resque-pool-worker[21162]: Starting worker portinari.local:21162:bar
88 resque-pool-manager[21134]: Reaped resque worker[21136] (status: 0) queues: bar
89 resque-pool-manager[21134]: Reaped resque worker[21135] (status: 0) queues: bar
90 resque-pool-worker[21176]: Starting worker portinari.local:21176:bar
91 resque-pool-worker[21178]: Starting worker portinari.local:21178:bar
92 resque-pool-manager[21134]: Reaped resque worker[21141] (status: 0) queues: foo
93 resque-pool-manager[21134]: Reaped resque worker[21138] (status: 0) queues: foo
94 resque-pool-worker[21180]: Starting worker portinari.local:21180:*
95 resque-pool-worker[21181]: Starting worker portinari.local:21181:foo
96 ^C>> exit
97 Bye!
98 Shutting down resque-pool-master 21134
99 resque-pool-manager[21134]: INT: immediate shutdown (graceful worker shutdown)
100 resque-pool-manager[21134]: manager finished
101 $ _
102
103 ## Command Line Interface
104
105 The CLI is a slightly customized Ripl shell (an IRB replacement)
106 started in context of a `Resque::Pool::Dynamic` instance. You can use
107 all Ruby you want, and call all the methods for controlling the resque
108 pool. A simple built-in help for the custom methods is included:
109
110 >> help
111 Known commands:
112 config - "WORKERS", as interpreted by the #parse_config_string method.
113 config_path - Path to the temporary config file.
114 exit - Finish work
115 has_log? - True if we have an open log file.
116 kill! - Send signal to a running resque-pool.
117 log - Open log of resque-pool process.
118 log.ff - Fast forward until end of file (see IO::Tail#backward).
119 log.rew - Rewind until beginning of file (see IO::Tail#forward).
120 log.rewind - Rewind until beginning of file (see IO::Tail#forward).
121 log.tail - Show last n lines of the file, or follow the file.
122 log.tail_to_eof - Print contents of the file until current end of file.
123 log.tail_until - Follow the file until a line matches regexp.
124 log_path - Logfile path.
125 pid - PID of the resque-pool process, or nil if it's not running.
126 pstree - Show child process tree by calling `pstree` system command.
127 reload - Reload resque-pool configuration.
128 start - Start resque-pool, showing startup logs.
129 start! - Fork a child process for resque-pool master.
130 status - Show current status.
131 stop - Stop running resque-pool, show shutdown logs.
132 stop! - Stop running resque-pool.
133 write_config - Write temporary configuration file.
134 For details, type: help command
135 >> help config
136 -------------------------------- Method: #config (Resque::Pool::Dynamic)
137 (Defined in: lib/resque/pool/dynamic/process.rb)
138 dynamic.config -> Hash
139 dynamic.config(opts) -> Hash
140 ------------------------------------------------------------------------
141 Note: Default configuration is taken from environment variable
142 "WORKERS", as interpreted by the #parse_config_string method.
143 Examples:
144 ---------
145 # config
146 #=> {"foo"=>1, "bar"=>2}
147 config :foo => 2, :baz => 7
148 #=> {"foo"=>2, "baz"=>7, "bar"=>2}
149 config :bar => 0
150 #=> {"foo"=>2, "baz"=>7}
151 Overloads:
152 ----------
153 ------------------------------------------------------------------------
154 dynamic.config -> Hash
155 ------------------------------------------------------------------------
156 Show current workers configuration
157 Returns:
158 --------
159 (Hash) -
160 ------------------------------------------------------------------------
161 dynamic.config(opts) -> Hash
162 ------------------------------------------------------------------------
163 Update workers configuration. If configuration has change and
164 resque-pool is running, it is reloaded.
165 Parameters:
166 -----------
167 (Hash) opts - Dictionary of worker process counts to update
168 (pass 0 or nil as value to delete all workers for a queue from pool)
169 Returns:
170 --------
171 (Hash) - Updated config
172 >> _
173
174 The built-in help is autogenerated from Yard documentation on methods;
175 if you're curious, look into the Rakefile for details.
176
177 ## Rake task documentation
178
179 $ bundle exec rake -D resque:pool:dynamic
180 rake resque:pool:dynamic
181 Starts a dynamic pool of Resque worker processes.
182
183 Variables:
184 WORKERS=worker spec - specify initial numbers of workers
185 (look below for format description)
186 NO_START=1 - don't start resque-pool master automatically
187 (default is to start)
188
189 Workers spec format:
190 queue_name=process_number[:queue_name=process_number[:...]]
191
192 queue_name can be whatever resque:work will accept as QUEUE=
193 parameter.
194
195 Example:
196 $ rake resque:pool:dynamic WORKERS=foo=1:bar=1:baz,quux,xyzzy=5:*=2
197 will start:
198 - one worker for queue 'foo'
199 - one worker for queue 'bar'
200 - five workers for queues baz,quux,xyzzy
201 - two workers for all queues
202
203 ## API and code structure
204
205 The Rake task, defined in `resque/pool/dynamic/tasks.rb`, is a
206 one-liner calling out to `Resque::Pool::Dynamic.shell`, which starts
207 the command line interface.
208
209 The CLI itself, defined in `resque/pool/dynamic/shell.rb`, is a thin
210 layer (8 lines of code + Ripl UI tweaks) over an instance of the
211 `Resque::Pool::Dynamic` class.
212
213 The workhorse `Resque::Pool::Dynamic` class is defined in
214 `resque/pool/dynamic.rb`. It can be easily instantiated in the code
215 independent from the CLI; the CLI commands are just instance methods
216 you can call then.
217
218 ### Ideas
219
220 Besides interactive, supervised work, `Resque::Pool::Dynamic` can be
221 used separately from the UI for dynamic, reactive management of
222 workers, implementing some kind of autoscaling and load distribution
223 logic.
224
225 It can be used during load tests to automatically optimize best number
226 of workers for maximum processing speed, while keeping load average
227 and memory or CPU usage goals.
228
229 It may also react to queue depths, e.g. temporarily reallocating idle
230 workers to overloaded queues or shutting them down to conserve memory,
231 or actively managing sudden short peak usage by stopping other workers
232 and switching them to peaking queue.
233
234 It may also simply allocate workers dynamically based on time of day,
235 to e.g. have more workers performing UI-related task in the day, which
236 increases perceived responsiveness, and switch them to process
237 long-running reports overnight when there are less users.
238
239 ## Detailed documentation and source code
240
241 - http://rubydoc.info/github/mpasternacki/resque-pool-dynamic
242 - http://github.com/mpasternacki/resque-pool-dynamic
Something went wrong with that request. Please try again.