Warn/disallow `Restart=always` without preventing `repeated too quickly` loop

[wallentx]

Component

systemd

Is your feature request related to a problem? Please describe

I've recently opened a few issues, and PRs for various projects to address failure conditions that are all related to these services using a systemd *.service file that they've configured by default with Restart=always, but have not included any StartLimitIntervalSec, StartLimitBurst, or RestartSec directives to prevent the service from entering an infinite Start request repeated too quickly loop.

I don't know if there's a commonly understood best-practice for creating service units that handle failures smartly. It just seems that some projects do, while others don't.

The problem in several of the situations I've experienced, and have tried to address, is that when this endless loop happens, it manifests in a way that doesn't leave the user with any obvious understanding as to what exactly is happening, such as just a blank unresponsive screen.

winesapOS/winesapOS#716

ubuntu/lightdm#340

https://gitlab.archlinux.org/archlinux/packaging/packages/openssh/-/issues/3

Only to help illustrate what a user might experience, I asked ChatGPT to talk me through the execution steps of the gdm3 .service unit, in a scenario where xorg.conf contains a value that will cause x11 to immediately fail 100% of the time:
https://chat.openai.com/share/ca3c9786-53f2-476e-8c21-17b4465c4327

Describe the solution you'd like

While I don't think that it is systemd's responsibility that projects use systems units in the most optimal way, I feel like a wandering lunatic roaming from project to project suggesting changes, and having to justify why this was an annoying issue that I've had to work around (such as the case with sshd.service for literally any device running Arch on my network, for whatever reason), and so I wanted to check upstream with the common piece around all this.

At a minimum, perhaps it can be agreed upon that using Restart=always without any supporting directives may possibly result in a negative outcome, as it assumes that restarting the service will magickally fix everything?

If so, maybe a check, and a warning when the user executed daemon-reload?

I don't want to get ahead of myself, and there might be something I'm misunderstanding, so I want to first verify that what I understand is accurate, and that my concern is valid. If so, I hoped to open a conversation about this to see it there is any sort of consensus that can be agreed upon.

Describe alternatives you've considered

Have some reasonable built-in default values for StartLimitIntervalSec, and StartLimitBurst that are used if Restart=always is specified.
Or maybe there ARE default values, and the presence of the repeated too quickly messages means that the service had exceeded those defaults, and systemd permanently put the service in a failed state that requires intervention to restart?
I'm not really clear on the behavior, but logs seem to indicate to me that there are repeated occurrences of repeated too quickly.

The systemd version you checked that didn't have the feature you are asking for

255

[YHNdnzj]

Or maybe there ARE default values, and the presence of the repeated too quickly messages means that the service had exceeded those defaults, and systemd permanently put the service in a failed state that requires intervention to restart?

This is already the case, i.e. if start limit is exceeded, no further auto-restart will be allowed. So I'm not really sure what this is about?

[stapelberg]

While the original report might not have been perfectly phrased, I think there is indeed substance to the report and would like to add my observations and specific suggestion.

I published a blog post titled “systemd: enable indefinite service restarts” where I share the one important bit of systemd knowledge that I apply to every single server system I maintain: how to make services restart indefinitely, instead of having systemd give up on them:

https://michael.stapelberg.ch/posts/2024-01-17-systemd-indefinite-service-restarts/

To summarize: the problem is that systemd’s default restart limits are chosen such that systemd gives up on restarting crash-looping services:

DefaultRestartSec=100ms
DefaultStartLimitIntervalSec=10s
DefaultStartLimitBurst=5

I don’t know if this was an intentional decision, or if the limits were chosen arbitrarily.

Either way, to give a specific suggestion, how about we change the default configuration to:

[Manager]
DefaultRestartSec=5s
DefaultStartLimitIntervalSec=0

With these settings, services will be restarted indefinitely, with a 5-second delay in between. This doesn’t seem too aggressive, and should result in transient failures eventually resolving themselves without human intervention.

What do you think?

[martinpitt]

I find the default behaviour quite appropriate; at least I oppose defaulting or even suggesting an indefinite unthrottled restart behaviour with the suggested approach. IMO it's bad advice, especially the bit with enabling it for all services. If you apply such a big blunt "retry harder" hammer, you should really know what you are doing..

If your service crashes 5 times in 10 seconds, it's just plain broken. Trying to restart it will hide it from systemctl --failed, obfuscate it in the logs, is a resource waste, possibly even a data loss (think databases or services which spew lots of files to the disk and drive it to ENOSPC), and make it less apparent that it needs fixing.

A possible compromise might be exponential back-off with Restart=always|on-failure, so that e.g. services which crash on external dependencies (it's always DNS, right?) try to recover every hour or so after having failed a dozen times. Would that suit your use case better?

[stapelberg]

Hey Martin 👋 Good to read from you, it’s been a long time :) Hope you’re doing well.

IMO it's bad advice, especially the bit with enabling it for all services. If you apply such a big blunt "retry harder" hammer, you should really know what you are doing..

Note that we’re only talking about changing the behavior for services that specify Restart=always|on-failure.

Trying to restart it will hide it from systemctl --failed,

Yeah, this is a good point that has also been pointed out in the discussion on Mastodon: Some people alert on failed systemd services, so if a service never fails, it won’t show up in their monitoring/alerting.

Personally, I prefer monitoring/alerting based on probing the application, not specific systemd units, but I agree that this is a difference in philosophy and that a crash-looping service should be discoverable as a problem in monitoring.

Maybe systemd could expose that the service is restarting, but still keep trying to start it?

In my Go appliance platform gokrazy, I’m exposing crashing services with a “restarting” status. For example, in this case, the frame buffer status display program is restarting because no monitor is connected:

A possible compromise might be exponential back-off with Restart=always|on-failure, so that e.g. services which crash on external dependencies (it's always DNS, right?) try to recover every hour or so after having failed a dozen times. Would that suit your use case better?

Slowing down restarts to a crawl — you suggest once per hour after backoff, though hopefully the exact upper limit will be configurable — sounds like a good improvement to the status quo to me.

[Werkov]

backoff, though hopefully the exact upper limit will be configurable ... improvement to the status quo to me.

That is the status quo.

[wallentx]

I've been really struggling to figure out how to better communicate what's in my head with this, but it seems that @stapelberg had the idea. Thanks for that.

I just watched this https://youtu.be/KfSF9Cmm73w and what is being described is the same problem that I was trying to describe.. but in this case, the problem in need of a solution is expanded to kernel panics, so a systemd solution might not cover those cases.

I think if I had to reword the issue that im describing, it would be something like:

Linux feels like it is missing some component that can:

• effectively catch failures that happen prior to the completion of, or during the user login process
• communicate those failures to the user
• provided some useful suggestion for remediation, if available

You could argue that none of this is the responsibility of systemd. I'm not actually sure where the responsibility of something like this might best be.
I don't know that the work being done here https://lore.kernel.org/dri-devel/2dc1b7c6-1743-4ddd-ad42-36f700234fbe@linux.intel.com/T/#u covers all of the situations I'm talking about.

I do still think that there could be some good done on the systemd side - if even just a mention in docs - to encourage someone creating a systemd service to be mindful of how their service fails, what happens when it fails, and to avoid the scenario where a "service restarted too quickly" loops for infinity.

[kevincox]

I'm also not satisfied here. RestartSteps and RestartMaxDelaySec are a nice step in the right direction but they never reset. So if I am OOO over Christmas and my service is restarting occasionally I'm going to start having unfortunately long downtimes.

I think the following would work well for me:

1. Add DefaultRestartSteps and DefaultRestartMaxDelaySec so that I can set this up system-wide.
2. Add RestartDelayResetSec and DefaultRestartDelayResetSec options to reset the restart delay once a service has been active for some period of time.

This way I can set up some reasonable default for my system (Maybe something like RestartSec=0, RestartSteps=1800, RestartMaxDelaySec=1h so that constantly crashing service don't burn CPU but try to recover from time-to-time) and then override it for specific services (you know that one service that is the whole point of the box like the HTTP server) to restart much more frequently forever.

Right now I really have no option but to just use a fixed restart interval with no restart limit. This isn't optimal but just about anything is better than giving up on any sort of auto-recovery and requiring an operator to urgently address the issue.

[Werkov]

I think the following would work well for me:

1. Add `DefaultRestartSteps` and `DefaultRestartMaxDelaySec` so that I can set this up system-wide.

2. Add `RestartDelayResetSec` and `DefaultRestartDelayResetSec` options to reset the restart delay once a service has been active for some period of time.

Just a pointer to systemd.unit(5):

Units also support a top-level drop-in with type.d/, where type may be e.g. "service" or "socket", that allows altering or adding to the settings of all corresponding unit files on the system.

The manpage has an example with OnFailure= handler but it should cover also your restarting default + specialization.

[alcohol]

Not sure if the following is relevant to the discussion or warrants a new issue. We have a rabbitmq message consumer that gracefully exits (with status code 0) after a defined period (3600s to be specific). We do this because it is a php process and php does not have the best track record when it comes to long running processes and garbage collection / memory leaks. However, these graceful exits also count towards the restart count (we use Restart=always). This basically prevents us from using the backoff mechanism (which we do want to use ideally). Several of the changes discussed in this topic could contribute towards a solution in this regard. I am also curious if it would make sense to perhaps make a distinction between graceful restarts (clean exit) and restarts as a result from failure/ab-normal exits?