Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 102 lines (91 sloc) 3.572 kB
99e49b4 @TwP Adding an example using Beanstalkd.
authored
1 # Preforking Beanstalkd job runner using Servolux.
2 #
3 # In this example, we prefork 7 processes each of which connect to our
4 # Beanstalkd queue and then wait for jobs to process. We are using a module so
77b9aea @lmarburger fixing typos
lmarburger authored
5 # that we can connect to the beanstalk queue before executing and then
99e49b4 @TwP Adding an example using Beanstalkd.
authored
6 # disconnect from the beanstalk queue after exiting. These methods are called
7 # exactly once per child process.
8 #
9 # A variation on this is to load source code in the before_executing method
77b9aea @lmarburger fixing typos
lmarburger authored
10 # and initialize an object that will process jobs. This is advantageous because
99e49b4 @TwP Adding an example using Beanstalkd.
authored
11 # now you can send SIGHUP to a child process and it will restart, loading your
12 # Ruby libraries before executing. Now you can do a rolling deploy of new
13 # code.
14 #
15 # def before_executing
16 # Kernel.load '/your/source/code.rb'
17 # @job_runner = Your::Source::Code::JobRunner.new
18 # end
19 # --------
20
21 require 'servolux'
22 require 'beanstalk-client'
23
24 module JobProcessor
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
25 # Open a connection to our beanstalk queue. This method is called once just
26 # before entering the child run loop.
99e49b4 @TwP Adding an example using Beanstalkd.
authored
27 def before_executing
40a9e36 @TwP adding in worker configuration options to the examples
authored
28 host = config[:host]
29 port = config[:port]
30 @beanstalk = Beanstalk::Pool.new(["#{host}:#{port}"])
99e49b4 @TwP Adding an example using Beanstalkd.
authored
31 end
32
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
33 # Close the connection to our beanstalk queue. This method is called once
34 # just after the child run loop stops and just before the child exits.
99e49b4 @TwP Adding an example using Beanstalkd.
authored
35 def after_executing
36 @beanstalk.close
37 end
38
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
39 # Close the beanstalk socket when we receive SIGHUP. This allows the execute
40 # thread to return processing back to the child run loop; the child run loop
41 # will gracefully shutdown the process.
42 def hup
43 @beanstalk.close if @job.nil?
44 @thread.wakeup
45 end
46
47 # We want to do the same thing when we receive SIGTERM.
48 alias :term :hup
49
99e49b4 @TwP Adding an example using Beanstalkd.
authored
50 # Reserve a job from the beanstalk queue, and processes jobs as we receive
51 # them. We have a timeout set for 2 minutes so that we can send a heartbeat
52 # back to the parent process even if the beanstalk queue is empty.
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
53 #
54 # This method is called repeatedly by the child run loop until the child is
55 # killed via SIGHUP or SIGTERM or halted by the parent.
99e49b4 @TwP Adding an example using Beanstalkd.
authored
56 def execute
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
57 @job = nil
58 @job = @beanstalk.reserve(120) rescue nil
59 if @job
60 $stdout.puts "[C] #{Process.pid} processing job #{@job.inspect}"
61 # ... do more processing here
99e49b4 @TwP Adding an example using Beanstalkd.
authored
62 end
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
63 rescue Beanstalk::TimedOut
64 ensure
65 @job.delete rescue nil if @job
99e49b4 @TwP Adding an example using Beanstalkd.
authored
66 end
67 end
68
69 # Create our preforking worker pool. Each worker will run the code found in
40a9e36 @TwP adding in worker configuration options to the examples
authored
70 # the JobProcessor module.
71 #
72 # The `:config` Hash is passed to each worker when it is created. The values
73 # here are available to the JobProcessor module. We use this config hash to pass
74 # the `:host` and `:port` where the beanstalkd server can be found.
75 #
76 # We set a timeout of 10 minutes for the worker pool. The child process
99e49b4 @TwP Adding an example using Beanstalkd.
authored
77 # must send a "heartbeat" message to the parent within this timeout period;
78 # otherwise, the parent will halt the child process.
79 #
80 # Our execute code in the JobProcessor takes this into account. It will wakeup
81 # every 2 minutes, if no jobs are reserved from the beanstalk queue, and send
82 # the heartbeat message.
83 #
84 # This also means that if any job processed by a worker takes longer than 10
85 # minutes to run, that child worker will be killed.
40a9e36 @TwP adding in worker configuration options to the examples
authored
86 pool = Servolux::Prefork.new \
87 :timeout => 600,
88 :module => JobProcessor,
89 :config => {:host => '127.0.0.1', :port => 11300}
99e49b4 @TwP Adding an example using Beanstalkd.
authored
90
91 # Start up 7 child processes to handle jobs
92 pool.start 7
93
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
94 # When SIGINT is received, kill all child process and then reap the child PIDs
95 # from the proc table.
96 trap('INT') {
97 pool.signal 'KILL'
98 pool.reap
99 }
99e49b4 @TwP Adding an example using Beanstalkd.
authored
100 Process.waitall
5d4e4d4 @TwP Documenting the examples for better comprehension and learning.
authored
101
Something went wrong with that request. Please try again.