God occasionally does not realise that a process has died and reports 0kb memory usage in log

[mloughran]

I'm running god on ubuntu 10.04

Ruby 1.8.7 (standard debian package)
God 0.11.0

I occasionally see an issue where the process is not running (accoring to ps), but god log reports something like the following

$ sudo god log my_process
Please wait...
I [2010-10-18 10:50:24]  INFO: my_process [trigger] process is running (ProcessRunning)
I [2010-10-27 16:14:17]  INFO: my_process [ok] memory within bounds [0kb, 0kb, 0kb, 0kb, 0kb] (MemoryUsage)
I [2010-10-27 16:14:37]  INFO: my_process [ok] memory within bounds [0kb, 0kb, 0kb, 0kb, 0kb] (MemoryUsage)
I [2010-10-27 16:14:57]  INFO: my_process [ok] memory within bounds [0kb, 0kb, 0kb, 0kb, 0kb] (MemoryUsage)
I [2010-10-27 16:15:17]  INFO: my_process [ok] memory within bounds [0kb, 0kb, 0kb, 0kb, 0kb] (MemoryUsage)
I [2010-10-27 16:15:37]  INFO: my_process [ok] memory within bounds [0kb, 0kb, 0kb, 0kb, 0kb] (MemoryUsage)

and god status reports that the process is up. Therefore the process does not get restarted - which is a Very Bad Thing.

The pid file in /var/run/god/my_process.pid reports a pid which doesn't correspond to any running process. Looking at the log files for the process which died, it appears that the process did not shut down cleanly.

Does anyone have any ideas what the issue could be, or how I could debug this when it happens again? Thanks.