Monitored daemon with multiple=false improperly runs multiple times

[washogren]

Hi!

I've run into a bug that's unfortunately a big issue for us. Basically, we have a daemon running, and we only want one instance of the daemon to run at once, and we run with monitoring enabled so that it is restarted automatically on error conditions.

We're running it with a command like the following:
Daemons.run(SERVICE_PATH, app_name: SERVICE_NAME, multiple: false, mode: :exec, monitor: true, log_output: true, dir: LOG_DIR, dir_mode: :normal, force_kill_waittime: 15, )

We end up in a state occasionally where we will end up with multiple instances of our service running at once, as well as multiple instances of the associated monitors.

I've narrowed this down to occurring when you delete the pid files for the service at the same time that you delete the pid file for the monitor, and then start the service again via the command line. Even though the files have been deleted, the monitor and service continues running as before.

Starting the service again via the command line creates a new monitor, which starts a new service, and we end up with a state with multiple monitors and multiple services running at the same time.

This could potentially be fixed by having the monitor watch occasionally to ensure that it's pid file still exists, and if it doesn't it could shut everything down. Another alternative could be to have the startup logic look for the process name in addition to the pid file in an attempt to catch this case.

Any thoughts? I'm happy to answer any questions!

[pgouv]

Its over 1y but I have similar issue. Ended up using monit for this