Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 91 lines (80 sloc) 3.212 kb
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
1 # Preforking Beanstalkd job runner using Servolux.
2 #
3 # In this example, we prefork 7 processes each of which connect to our
4 # Beanstalkd queue and then wait for jobs to process. We are using a module so
77b9aea Larry Marburger fixing typos
lmarburger authored
5 # that we can connect to the beanstalk queue before executing and then
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
6 # disconnect from the beanstalk queue after exiting. These methods are called
7 # exactly once per child process.
8 #
9 # A variation on this is to load source code in the before_executing method
77b9aea Larry Marburger fixing typos
lmarburger authored
10 # and initialize an object that will process jobs. This is advantageous because
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
11 # now you can send SIGHUP to a child process and it will restart, loading your
12 # Ruby libraries before executing. Now you can do a rolling deploy of new
13 # code.
14 #
15 # def before_executing
16 # Kernel.load '/your/source/code.rb'
17 # @job_runner = Your::Source::Code::JobRunner.new
18 # end
19 # --------
20
21 require 'servolux'
22 require 'beanstalk-client'
23
24 module JobProcessor
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
25 # Open a connection to our beanstalk queue. This method is called once just
26 # before entering the child run loop.
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
27 def before_executing
28 @beanstalk = Beanstalk::Pool.new(['localhost:11300'])
29 end
30
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
31 # Close the connection to our beanstalk queue. This method is called once
32 # just after the child run loop stops and just before the child exits.
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
33 def after_executing
34 @beanstalk.close
35 end
36
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
37 # Close the beanstalk socket when we receive SIGHUP. This allows the execute
38 # thread to return processing back to the child run loop; the child run loop
39 # will gracefully shutdown the process.
40 def hup
41 @beanstalk.close if @job.nil?
42 @thread.wakeup
43 end
44
45 # We want to do the same thing when we receive SIGTERM.
46 alias :term :hup
47
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
48 # Reserve a job from the beanstalk queue, and processes jobs as we receive
49 # them. We have a timeout set for 2 minutes so that we can send a heartbeat
50 # back to the parent process even if the beanstalk queue is empty.
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
51 #
52 # This method is called repeatedly by the child run loop until the child is
53 # killed via SIGHUP or SIGTERM or halted by the parent.
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
54 def execute
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
55 @job = nil
56 @job = @beanstalk.reserve(120) rescue nil
57 if @job
58 $stdout.puts "[C] #{Process.pid} processing job #{@job.inspect}"
59 # ... do more processing here
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
60 end
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
61 rescue Beanstalk::TimedOut
62 ensure
63 @job.delete rescue nil if @job
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
64 end
65 end
66
67 # Create our preforking worker pool. Each worker will run the code found in
68 # the JobProcessor module. We set a timeout of 10 minutes. The child process
69 # must send a "heartbeat" message to the parent within this timeout period;
70 # otherwise, the parent will halt the child process.
71 #
72 # Our execute code in the JobProcessor takes this into account. It will wakeup
73 # every 2 minutes, if no jobs are reserved from the beanstalk queue, and send
74 # the heartbeat message.
75 #
76 # This also means that if any job processed by a worker takes longer than 10
77 # minutes to run, that child worker will be killed.
78 pool = Servolux::Prefork.new(:timeout => 600, :module => JobProcessor)
79
80 # Start up 7 child processes to handle jobs
81 pool.start 7
82
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
83 # When SIGINT is received, kill all child process and then reap the child PIDs
84 # from the proc table.
85 trap('INT') {
86 pool.signal 'KILL'
87 pool.reap
88 }
99e49b4 Tim Pease Adding an example using Beanstalkd.
authored
89 Process.waitall
5d4e4d4 Tim Pease Documenting the examples for better comprehension and learning.
authored
90
Something went wrong with that request. Please try again.