fix(service): Change systemd's KillMode to "mixed"

[knollet]

Currently all processes created by Telegraf are killed as-soon-as a the service is stopped. This triggers some errors in Telegraf as its child processes died unexpectedly. By changing the KillMode to mixed Telegraf is given a chance to shut-down its child-processes cleanly while keeping the property of a clean exit if it fails.

fixes #13842

[srebhan]

Looks good to me. Thanks for the quick fix @knollet!

[powersj]

Thank you!

[Hipska]

I was wondering for a while why telegraf wasn't shutting down gracefully anymore.

It appears because of the added TimeoutStopSec parameter which gives telegraf only 5 seconds to shut down. This is problematic when having inputs that take more time than that to complete their interval (e.g. snmp with slow or very big machines can take more than a minute to complete). So when such a service is running when reloading or stopping telegraf, systemd will give telegraf only 5 seconds to complete input gather and flush the outputs and then kill it having all unsent metrics with it.

What is the reason behind this 5s value?

[knollet]

I am good with a larger value.

[Hipska]

Is the systemd default not good enough? Why was the directive added?

[knollet]

KillMode=mixed was important to me, else systemd kills telegraf's subprocesses.
The systemd documentation references the TimeoutStopSec as the timeout after which it, again, goes about killing telegraf's subprocesses, so I wanted to define a value.
But I think that can be omitted. The default seems to be 90s.

[powersj]

@knollet was there a reason you set the TimeoutStopSec=5 rather than keep the default? Would you be good with us removing that option and keep to the default?

[knollet]

I am good with the default, yes. I added it because the documentation referenced it. But having a closer look, it seems not to be necessary to explicitly set it.