Orphaned process issue

[ddollar]

Sometimes processes are not shut down correctly. Seems to be related to whether or not there is a $ in the Procfile entry.

[kevinburke]

I can add another case to the pile... my Procfile is

web: lein run -m com.twilio.smsauth.core

and running foreman start, with or without a hardcoded $PORT in my env leaves a Java subprocess running. I did not have an issue before about two weeks ago, but not sure what happened. Reproduced with Ruby 1.8.7 and 1.9.2, Foreman 0.26.1 and Snow Leopard.

[bloudermilk]

Any chance this issue will get some attention soon?

[ddollar]

Added some detail in #99.

[be9]

foreman 0.25.0 is not killing thin process under Ruby 1.9.3-p0. Happens 100% of the time.

[ddollar]

I just released foreman 0.28.0.pre1 that may fix this. Please try it and let me know if it works for you.

$ gem install foreman --pre

[be9]

Thank you, David, now it correctly kills everything. However I noticed something. Here's a foreman run log for my app:

https://gist.github.com/258cf4406a9993310322

Some messages printed by processes are only output to console after I press Ctrl-C. With previous versions of foreman they were right out there.

[ddollar]

Try adding this to your app, as high up in the call stack as possible:

$stdout.sync = true

[brainopia]

v0.28 fixed it only partially, previously commands were run through ruby exec method, but if command contained *?{}[]<>()~&|\\$;'"\n` then it would fork and run sh -c with this command, therefore incorrect process was killed by foreman.

Since v0.28 shell exec function is used, but it does not support complex commands (with pipes and redirections and several commands). There are already couple of open issues related to that.

I've fixed this problem in #125 by using ruby exec method again, but tracking whole process tree with group process id and closing it when needed. As additional feature now foreman supports not only running complex commands but closing it also even if they contain subshells.

[ddollar]

This should be fixed in the latest foreman. Complex commands can't be used, but if you need something like that you can always create a bin/web wrapper script to do some crazy things for your web process.

[betamatt]

Unfortunately, fixing this issue has broken a common way of running thin:
web: thin -p $PORT -e ${RACK_ENV-development} start

The rack env default value is no longer interpolated and rails crashes trying to load the "${RACK_ENV-development}" environment.

[ddollar]

This is my solution to this problem:

$ cat Procfile
web: bin/web

$ cat bin/web
#!/bin/bash

PORT=${PORT:-5000}
RACK_ENV=${RACK_ENV:-development}
ROOT=$(dirname $(dirname $0))

export RUBYOPT="-I$ROOT/lib -rstdout"

if [[ "${RACK_ENV}" == "development" ]]; then
  exec bundle exec shotgun -p $PORT
else
  exec bundle exec thin start -p $PORT
fi

$ cat lib/stdout.rb
$stdout.sync = true

[betamatt]

Doesn't that result in the same issues this bug was originally addressing? Let's say you have web and queue, with web being started by bin/web.sh as described. If for some reason queue fails to load, foreman will kill both the web and queue pids. The thin instance started by web.sh, being on another pid, ends up orphaned.

$ cat Procfile 
web: ./web.sh
queue:  bundle exec rake jobs:work RAILS_ENV=${RACK_ENV-development}

$ cat web.sh 
#!/bin/sh
thin -p $PORT -e ${RACK_ENV-development} start

$ bundle exec foreman start
14:39:04 web.1     | started with pid 82767
14:39:04 queue.1   | started with pid 82768
14:39:09 web.1     | >> Using rack adapter
14:39:11 queue.1   | rake aborted!
14:39:11 queue.1   | ${RACK_ENV-development} database is not configured
14:39:11 queue.1   | 
14:39:11 queue.1   | Tasks: TOP => jobs:work => environment
14:39:11 queue.1   | (See full trace by running task with --trace)
14:39:11 queue.1   | process terminated
14:39:11 system    | sending SIGTERM to all processes
14:39:11 system    | sending SIGTERM to pid 82767
14:39:11 system    | sending SIGTERM to pid 82767

$ ps
  PID TTY           TIME CMD
    ...
82769 ttys006    0:08.20 ruby /Users/matt/.rvm/gems/ruby-1.9.3-p0/bin/thin -p 5000 -e development start

[ddollar]

The key is having exec in your bin/web file. This replaces the existing process with the new process rather than wrapping it.

[betamatt]

Whoops, right.

I'd still argue that this feels really complicated for something that's not
exactly an edge case. Everyone running rails [edit: everyone running thin] is going to need to do this or
force their developers to create a .env file defining RACK_ENV.

Is there some simple way of defaulting the environment that I'm not aware
of?

On Tue, Jan 17, 2012 at 3:06 PM, David Dollar <
reply@reply.github.com

wrote:

The key is having "exec" in your bin/web file. This replaces the
existing process with the new process rather than wrapping it.

---

Reply to this email directly or view it on GitHub:
#94 (comment)

[ddollar]

Agreed that it's a bit more complicated to use. Being able to do ${RACK_ENV:-development} in your Procfile means that the Procfile runner needs to be able to interpret and run bash, which is also complicated and prone to failure. The orphaned process issue here was because the "automatic" bash interpreter that Ruby was using internally to parse its exec command lines was swallowing signals.

I'm certainly open to suggestions to improve the user experience while maintaining correctness.

[jzinn]

If you have Foreman in your Gemfile, you may want to alter the export RUBYOPT line from #94 (comment) to

export RUBYOPT="-I$ROOT/lib -rstdout${RUBYOPT:+ $RUBYOPT}"

to prevent the to_specs: Could not find bundler (>= 0) error from #243.