Support optional daemonization

[hassila]

What are your thoughts of supporting daemonization as part of this framework during startup? Useful for background services and looking at the proposed server side frameworks this looks like a natural place.

[hassila]

Or to paraphrase that, I’d be happy to work on a PR in that case.

[tomerd]

hi @hassila thanks for bringing this up. always happy to take ideas and PRs. The library in its current form offers a way to block the main thread waiting on a signal. what do you have in mind beyond that?

[hassila]

Thanks @tomerd, basically I started reviewing what facilities that are in place for building backend services (that are also portable between at least macOS and Linux) - having previous experience with system originally deployed on SysV OS:s (Solaris, HP-UX) which then were subsequently ported to MacOS / Linux, "daemonization" during startup was built-in as best practice (otherwise you would e.g. quit a manually started process when the controlling terminal was closed or logging out...).

It turns out after some research of current state-of-the-art that most of those practices aren't directly applicable today for deployment scenarios with launchd/systemd which provides a more robust foundation for deploying services (I guess that dates me.. :-)) - but this is mostly relevant for production system deployments using those facilities.

There is a decent writeup here:
https://man7.org/linux/man-pages/man7/daemon.7.html
and
https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html

I still think it might be useful during the development cycle to have an option to turn on old-style daemonization (maybe I'm missing something, but it doesn't seem too convenient to use launchd/systemd during typical workflows), that is basically the SySV approach outlined in the daemon7 manage above (disassociation of controlling terminal, etc, etc, basically optionally properly backgrounding the process when started from the command line). I was thinking of then adding that as an option to ServiceLifecycle.Configuration similar to how backtrace support is done.

For Linux, as a 'new style daemon' it may make sense to consider adding support for one additional point:
9. If applicable, a daemon should notify the init system about startup completion or status updates via the sd_notify(3) interface.

I'm not sure there is a similar interface for macOS, but it seems reasonable that the service lifecycle framework would support sd_notify(3) behind the scenes on Linux at least.

Additionally, in addition to SIGHUP/SIGTERM it would be nice to add support for SIGUSR1/SIGUSR2 out of the box, we've used those previously for service-specific tweaks (drop/rebuild caches, tweak debug levels, dump internal state, ...) which is sometimes convenient as an escape-hatch during development - as this framework provides signal handling in general it would be nice to support those too out of the box.

[Jean-Daniel]

It turns out after some research of current state-of-the-art that most of those practices aren't directly applicable today for deployment scenarios with launchd/systemd which provides a more robust foundation for deploying services (I guess that dates me.. :-)) - but this is mostly relevant for production system deployments using those facilities.

launchd/systemd is current state-of-the-art for desktop, but on the server side, the modern way is containerisation which imply that the running service has PID 1 and must not demonize.

I still think it might be useful during the development cycle to have an option to turn on old-style daemonization

What are you thinking about here ? AFAIK, most of the time, to develop such service, you run them in foreground to be able to debug them.

[Lukasa]

To add to the voices here, I'm +1 on not adding support for old-school daemonization. The unquestionable current best practice is that the difference between daemon and foreground process is only about how you were launched, and it's not for you to try to "escape" your parent to be re-parented to PID 1. If SysV needs you to do things the old-school way, a wrapper binary can always do this by performing the double-fork and then just execing your binary.

What I do think we should consider is whether there are specific behaviours in launchd, systemd, and friends that we should add support for. Of particular value would be asking whether there are particular signals used by these systems for instructions like "reload your config" or "shutdown gracefully", and providing those as built-in hooks on appropriate platforms.

[hassila]

What I do think we should consider is whether there are specific behaviours in launchd, systemd, and friends that we should add support for. Of particular value would be asking whether there are particular signals used by these systems for instructions like "reload your config" or "shutdown gracefully", and providing those as built-in hooks on appropriate platforms.

That seems to be supported already with SIGHUP/SIGTERM/SIGINT.

[hassila]

It turns out after some research of current state-of-the-art that most of those practices aren't directly applicable today for deployment scenarios with launchd/systemd which provides a more robust foundation for deploying services (I guess that dates me.. :-)) - but this is mostly relevant for production system deployments using those facilities.

launchd/systemd is current state-of-the-art for desktop, but on the server side, the modern way is containerisation which imply that the running service has PID 1 and must not demonize.

That is fine for most deployments, true - I'm (too) used to systems where containerisation overhead wouldn't be acceptable for the end users, so we often would be running on "bare metal".

I still think it might be useful during the development cycle to have an option to turn on old-style daemonization

What are you thinking about here ? AFAIK, most of the time, to develop such service, you run them in foreground to be able to debug them.

It would typically be a handful of services being codeveloped, typically one would run the stable ones in the background (using a simple startup script locally) and run the one under development in debugger in foreground when needed.

Probably containerisation for development would be fine, will read up on it, thanks.

Still think USR1/USR2 would be nice to add though even if using containers.

[hassila]

To add to the voices here, I'm +1 on not adding support for old-school daemonization. The unquestionable current best practice is that the difference between daemon and foreground process is only about how you were launched, and it's not for you to try to "escape" your parent to be re-parented to PID 1. If SysV needs you to do things the old-school way, a wrapper binary can always do this by performing the double-fork and then just execing your binary.

Ok, fair enough. For the record it's not about supporting SysV, that was just background info to explain where I was coming from to provide context. Main point was to make such a manually started process immune to being stopped if one accidentally closes the terminal window for example. I take it you are arguing for using a container then?

[Lukasa]

I'm mostly arguing for doing the simple thing.

One of the reasons the double-fork fell out of fashion is that it forces all executions of the binary to be parented on PID 1. This essentially means the binary can never be run other than as a daemon in this way. This makes it hard to integrate into modern init systems like launchd and systemd if you wanted to use them, and it makes it harder to develop the binary because it always forces itself into the background. Finally, it's incompatible with the container model, where usually the process in question becomes PID 1. Essentially the old-school daemonization strategy is inflexible: if you do this you always only do this.

If you want to avoid stopping a regular process when your terminal exits, there are ways to do that. For example, nohup & is a common solution.

[hassila]

I'd object to "always" as the title of this case is "optional", implying one would choose to enable it from the command line when starting (and it's also about redirecting stdout/stderr etc). So to integrate into modern init systems one would simply not choose to set this optional flag. I definitely didn't argue to change the default behaviour to a hardcoded daemonization, but rather wanted to understand if it would be a useful option for anyone else - if not, we'll close this.

I still think supporting USR1/USR2 would be nice though.

[Jean-Daniel]

The question is do we need to built-in daemonize(3) capabilities in the framework.

I don't think there is much value to add this capability, and it will bring more troubles than it helps.

Many macOS subsystems are mach-port based and so not fork safe (mach-port are not inherited by child process).

I'm not even sure the obj-c runtime (used on Apple Swift) is fork-safe. (it wasn't not so long ago).

While the situation may be simpler on Linux, I think that using a tool like daemonize still remain safer and a perfectly valid solution for integrating a service in an old-fashion service manager.

[hassila]

Ok. Added a separate issue for supporting user defined signals:

#81