Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add deferred signal-handling (fixes #332). #334

Merged
merged 2 commits into from

3 participants

@ged
ged commented

This uses a thread-local queue and a self-pipe so the code in the
trap blocks is properly re-entrant.

For details, see:

@ddollar
Owner

This seems like an alternative solution to #333. The reentrancy here seems desirable, especially since these signals tend to propagate.

@ged
ged commented

Yes, IMO introducing more threads is probably not the way to solve reentrancy problems.

ged added some commits
@ged ged Add deferred signal-handling (fixes #332).
This uses a thread-local queue and a self-pipe so the code in the
trap blocks is properly re-entrant.

For details, see:

  * http://cr.yp.to/docs/selfpipe.html
  * http://blog.rubybestpractices.com/posts/ewong/016-Implementing-Signal-Handlers.html
5ab08c6
@ged ged Try to allow children to shut down gracefully
Since signals will no longer be handled once foreman goes into
`terminate_gracefully`, default signal handlers are restored so as
not to cause it to get stuck in an unTERMable state.

This necessitates not using the process group for signalling
except as a last resort, as foreman itself will receive the signals
it sends. This splits `killall` into two methods, one which
signals only processes foreman itself has started, and one which
signals all processes in the process group to try to clean up
more aggressively, and then reworks `terminate_gracefully` to use
them.
1691883
@ged
ged commented

It looks like the Travis build is failing because this patch restores the default signal handlers at the start of shutdown (as the queue won't be read again), and killing a process group with kill( "-#{signal}", Process.getpgrp ) sends the TERM to the sending process as well, so it terminates the foreman process.

A better behavior would be to split killing children gracefully from killing the whole process group so you can try the first, and then fall back to the second in the timeout. The second commit does this.

@ged
ged commented

The only Travis failure is under 2.0, and is an unrelated issue fixed by #335.

@crcastle

I can haz merge? (post #335 merge to make Travis pass)

@ddollar ddollar merged commit 9fe7ddb into from
@ged ged deleted the branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 4, 2013
  1. @ged

    Add deferred signal-handling (fixes #332).

    ged authored
    This uses a thread-local queue and a self-pipe so the code in the
    trap blocks is properly re-entrant.
    
    For details, see:
    
      * http://cr.yp.to/docs/selfpipe.html
      * http://blog.rubybestpractices.com/posts/ewong/016-Implementing-Signal-Handlers.html
  2. @ged

    Try to allow children to shut down gracefully

    ged authored
    Since signals will no longer be handled once foreman goes into
    `terminate_gracefully`, default signal handlers are restored so as
    not to cause it to get stuck in an unTERMable state.
    
    This necessitates not using the process group for signalling
    except as a last resort, as foreman itself will receive the signals
    it sends. This splits `killall` into two methods, one which
    signals only processes foreman itself has started, and one which
    signals all processes in the process group to try to clean up
    more aggressively, and then reworks `terminate_gracefully` to use
    them.
This page is out of date. Refresh to see the latest.
Showing with 118 additions and 9 deletions.
  1. +118 −9 lib/foreman/engine.rb
View
127 lib/foreman/engine.rb
@@ -9,6 +9,10 @@
class Foreman::Engine
+ # The signals that the engine cares about.
+ #
+ HANDLED_SIGNALS = [ :TERM, :INT, :HUP ]
+
attr_reader :env
attr_reader :options
attr_reader :processes
@@ -33,6 +37,16 @@ def initialize(options={})
@processes = []
@running = {}
@readers = {}
+
+ # Self-pipe for deferred signal-handling (ala djb: http://cr.yp.to/docs/selfpipe.html)
+ reader, writer = create_pipe
+ reader.close_on_exec = true if reader.respond_to?(:close_on_exec)
+ writer.close_on_exec = true if writer.respond_to?(:close_on_exec)
+ @selfpipe = { :reader => reader, :writer => writer }
+
+ # Set up a global signal queue
+ # http://blog.rubybestpractices.com/posts/ewong/016-Implementing-Signal-Handlers.html
+ Thread.main[:signal_queue] = []
end
# Start the processes registered to this +Engine+
@@ -41,10 +55,7 @@ def start
# Make sure foreman is the process group leader.
Process.setpgrp unless Foreman.windows?
- trap("TERM") { puts "SIGTERM received"; terminate_gracefully }
- trap("INT") { puts "SIGINT received"; terminate_gracefully }
- trap("HUP") { puts "SIGHUP received"; terminate_gracefully } if ::Signal.list.keys.include? 'HUP'
-
+ register_signal_handlers
startup
spawn_processes
watch_for_output
@@ -53,6 +64,74 @@ def start
shutdown
end
+ # Set up deferred signal handlers
+ #
+ def register_signal_handlers
+ HANDLED_SIGNALS.each do |sig|
+ if ::Signal.list.include? sig.to_s
+ trap(sig) { Thread.main[:signal_queue] << sig ; notice_signal }
+ end
+ end
+ end
+
+ # Unregister deferred signal handlers
+ #
+ def restore_default_signal_handlers
+ HANDLED_SIGNALS.each do |sig|
+ trap(sig, :DEFAULT) if ::Signal.list.include? sig.to_s
+ end
+ end
+
+ # Wake the main thread up via the selfpipe when there's a signal
+ #
+ def notice_signal
+ @selfpipe[:writer].write_nonblock( '.' )
+ rescue Errno::EAGAIN
+ # Ignore writes that would block
+ rescue Errno::EINT
+ # Retry if another signal arrived while writing
+ retry
+ end
+
+ # Invoke the real handler for signal +sig+. This shouldn't be called directly
+ # by signal handlers, as it might invoke code which isn't re-entrant.
+ #
+ # @param [Symbol] sig the name of the signal to be handled
+ #
+ def handle_signal(sig)
+ case sig
+ when :TERM
+ handle_term_signal
+ when :INT
+ handle_interrupt
+ when :HUP
+ handle_hangup
+ else
+ system "unhandled signal #{sig}"
+ end
+ end
+
+ # Handle a TERM signal
+ #
+ def handle_term_signal
+ puts "SIGTERM received"
+ terminate_gracefully
+ end
+
+ # Handle an INT signal
+ #
+ def handle_interrupt
+ puts "SIGINT received"
+ terminate_gracefully
+ end
+
+ # Handle a HUP signal
+ #
+ def handle_hangup
+ puts "SIGHUP received"
+ terminate_gracefully
+ end
+
# Register a process to be run by this +Engine+
#
# @param [String] name A name for this process
@@ -98,11 +177,11 @@ def load_env(filename)
end
end
- # Send a signal to all processesstarted by this +Engine+
+ # Send a signal to all processes started by this +Engine+
#
# @param [String] signal The signal to send to each process
#
- def killall(signal="SIGTERM")
+ def kill_children(signal="SIGTERM")
if Foreman.windows?
@running.each do |pid, (process, index)|
system "sending #{signal} to #{name_for(pid)} at pid #{pid}"
@@ -113,6 +192,21 @@ def killall(signal="SIGTERM")
end
else
begin
+ Process.kill signal, *@running.keys unless @running.empty?
+ rescue Errno::ESRCH, Errno::EPERM
+ end
+ end
+ end
+
+ # Send a signal to the whole process group.
+ #
+ # @param [String] signal The signal to send
+ #
+ def killall(signal="SIGTERM")
+ if Foreman.windows?
+ kill_children(signal)
+ else
+ begin
Process.kill "-#{signal}", Process.getpgrp
rescue Errno::ESRCH, Errno::EPERM
end
@@ -277,8 +371,22 @@ def watch_for_output
Thread.new do
begin
loop do
- io = IO.select(@readers.values, nil, nil, 30)
+ io = IO.select([@selfpipe[:reader]] + @readers.values, nil, nil, 30)
+
+ begin
+ @selfpipe[:reader].read_nonblock(11)
+ rescue Errno::EAGAIN, Errno::EINTR => err
+ # ignore
+ end
+
+ # Look for any signals that arrived and handle them
+ while sig = Thread.main[:signal_queue].shift
+ self.handle_signal(sig)
+ end
+
(io.nil? ? [] : io.first).each do |reader|
+ next if reader == @selfpipe[:reader]
+
if reader.eof?
@readers.delete_if { |key, value| value == reader }
else
@@ -305,13 +413,14 @@ def watch_for_termination
def terminate_gracefully
return if @terminating
+ restore_default_signal_handlers
@terminating = true
if Foreman.windows?
system "sending SIGKILL to all processes"
- killall "SIGKILL"
+ kill_children "SIGKILL"
else
system "sending SIGTERM to all processes"
- killall "SIGTERM"
+ kill_children "SIGTERM"
end
Timeout.timeout(options[:timeout]) do
watch_for_termination while @running.length > 0
Something went wrong with that request. Please try again.