Daemonized remote process connected to STDOUT/STDERR causes sshkit to hang in loop

[yaauie]

I have a command on my remote server that daemonizes without disconnecting the output (using ruby's Process.daemon(true, true)), and continues writing to STDOUT after being daemonized. For demonstration purposes, here is my script:

# usr/bin/env ruby
# encoding: utf-8

puts "configurating..."
# error-prone stuff here, so I want its output
puts "configurated"
puts "daemonizing..."
Process.daemon(true, true) # the process returns status of 0 here.
puts "daemonized."
loop do
  sleep 1
  puts 'working...'
end

Once the process exits and sshkit gets an exit status code, I would expect it to stop paying attention to the command, but it doesn't.

 INFO [417676f7] Running RBENV_ROOT=/Users/yaauie/.rbenv RBENV_VERSION=1.9.3-p448 /Users/yaauie/.rbenv/bin/rbenv exec bundle exec do_work on 127.0.0.1
DEBUG [417676f7] Command: cd /Users/yaauie/worker-installation/current && ( RBENV_ROOT=~/.rbenv RBENV_VERSION=1.9.3-p448 RBENV_ROOT=/Users/yaauie/.rbenv RBENV_VERSION=1.9.3-p448 /Users/yaauie/.rbenv/bin/rbenv exec bundle exec do_work )
DEBUG [417676f7]  configurating...
DEBUG [417676f7]  configurated
DEBUG [417676f7]  daemonizing...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  daemonized.
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...
 INFO [417676f7] Finished in 0.607 seconds with exit status 0 (successful).
DEBUG [417676f7]  working...

This is the root cause behind 9z0b3t1c/capistrano-resque#80

[leehambley]

Interesting idea the Ruby gem has of "daemonized", please see the discussion of what daemonization really means at capistrano/capistrano#966.

[damphyr]

@leehambley I know it's really the fault of daemonize but this is something that keeps happening and there's a lot of bad behaved programs out there (I had to get one of the devs in my team to fix exactly this behaviour yesterday - long sermon on proper *ix daemonization inclusive.
Would a patch that drops out if an exit code is detected be worth it you think? We have a nice reproducible case as a test here.

[leehambley]

Would a patch that drops out if an exit code is detected be worth it you think? We have a nice reproducible case as a test here.

It would depend on what the patched ended up looking like, and how well it was functionally tested, based on the capistrano/vm. But, in theory yes.

I spent the last few years trying to find a knowledge war about badly behaved programs, and I'm not keen to do the same again. I also rarely see the use-case for starting daemons via Cap. (Surely, you'd want them monitored, and started by the system, and you would instruct the system to start them, using monit, god, bluepill, initd, runit, daemontools, initd, etc?)

But the long running badly behaved daemons argument has been ongoing for 20 years in linux, and people are still doing it wrong.

[leehambley]

I just checked the docs for Process.daemon, you are even using it incorrectly:

daemon() → 0 click to toggle source
daemon(nochdir=nil,noclose=nil) → 0
Detach the process from controlling terminal and run in the background as system daemon. Unless
the argument nochdir is true (i.e. non false), it changes the current working directory to the root
(“/”). Unless the argument noclose is true, daemon() will redirect standard input, standard output
and standard error to /dev/null. Return zero on success, or raise one of Errno::*.

Issue closed, people need to RTFM.