Terminating multiple times

[brandonbloom]

When terminating, I see output like this:

18:33:49 system    | terminating
18:33:49 system    | killing worker in pid 29580
18:33:49 system    | killing client in pid 29587
18:33:49 system    | killing api in pid 29582
18:33:49 system    | killing nginx in pid 29577
18:33:49 system    | killing solr in pid 29578
18:33:49 client.1  | process terminated
18:33:49 system    | terminating
18:33:49 system    | killing worker in pid 29580
18:33:49 system    | killing api in pid 29582
18:33:49 system    | killing nginx in pid 29577
18:33:49 system    | killing solr in pid 29578

This is a result of the forking/killing/waiting behavior in foreman/engine.rb, where child processes (not just the master process) are also killing their siblings.

Generally, this isn't a big deal. However, some applications treat the first SIGTERM as "start shutting down", but treat the second SIGTERM as "OMG SHUT DOWN QUICKLY GO GO GO GO GO". I've been playing with this a bit and I am seeing cases where Thin, Nginx, and others hold open a port (in a sub-sub-process?) because they shutdown to aggressively.

I'd be willing to take a crack at a patch if anyone thinks they know what the correct solution is.

Also, as a workaround, I'm about to attempt a wrapper process that eats all but the first SIGTERM.

[brandonbloom]

Here's the workaround script that I'm using now:

$terminating = false

def terminate
  Process.kill :TERM, $pid unless $terminating
  $terminating = true
end

trap('SIGHUP') { terminate }
trap('TERM') { terminate }
trap('INT') { terminate }

$pid = fork

exec ARGV.join(' ') unless $pid

Process.wait

[ddollar]

I just pushed up 0.18.0 that tightens up the signals coming out. when it comes time to terminate, all processes should receive a SIGTERM followed 3 seconds later by a SIGKILL if they're still alive