systemd should restart icinga2 automatically

[ekeih]

Usually icinga2 is a mission critical part of infrastructure and should be online at all times. In case of a recoverable crash (e.g. OOM) icinga2 should be restarted automatically. Of course it would be useful to monitor/log/alert the automatic restarts to investigate them afterwards.

Expected Behavior

Systemd should try to restart icinga2 in case of an unexpected crash.

Current Behavior

If icinga2 crashes it will stay in failed state until manual intervention.

Possible Solution

We currently override the systemd service with a drop-in file in /etc/systemd/system/icinga2.service.d/override.conf which contains:

[Service]
Restart=always
RestartSec=1
StartLimitInterval=10
StartLimitBurst=3

The place for a solution is https://github.com/Icinga/icinga2/blob/master/etc/initsystem/icinga2.service.cmake#L5. It would be necessary to include the changes in the packaging repository, e.g. https://github.com/Icinga/icinga-packaging/blob/deb/release/icinga2/ubuntu/debian/icinga2-common.icinga2.service#L5.

Steps to Reproduce (for bugs)

1. Start icinga2 with systemd.
2. Kill the icinga2 process, e.g. killall icinga2
3. systemd does not restart icinga

Your Environment

• Version used (icinga2 --version): r2.7.1-1
• Operating System and version: Ubuntu 16.04.3 LTS with systemd

I guess different people have a different opinion about automatic service restarts, so we can use this ticket for a discussion. If you approve the changes I would be ready to implement it and to provide a pull request.

[dgoetz]

I am against automatic restart as it is only necessary to hide application or environment faults like segfault or OOM. Also defaults for how long to wait for a restart, how long a start up can take and so on can hurt much more environments than they are useful. Furthermore it only affects some supported platforms, so there is a different behavior between newer and older Linux distributions and Windows.

But how you do it is a good way and generally sane defaults if someone wants this. So while I down vote for including this into official packages, I would like this to be added in documentation for those looking for.

[dnsmichi]

This should be handled by packages themselves, e.g. Ubuntu where a restart possibly has other implications than on a RHEL based system. Feel free to send in a PR for the docs.