How to set Restart=always for a socket unit?

[lahwaacz]

Even socket units may fail, e.g. due to an out-of-memory situation, which may trigger errors like

systemd[1]: sshd.socket: Failed to fork off accept stub process: Cannot allocate memory

(The unit comes from Arch Linux openssh package.)

Since there is no "nice way to fix the root cause", setting Restart=always for the unit is needed to prevent losing access to headless hosts with difficult physical access, but that clause is valid only for services. Is there another way to have socket units always restart on failure?

[mailinglists35]

what is the workaround until a native solution is provided? create a service to check and restart the socket unit?

[travier]

According to https://bugzilla.redhat.com/show_bug.cgi?id=2025716#c13, we should use TriggerLimitIntervalSec=0 (or a much higher value than the default) for those socket units. From systemd.socket

TriggerLimitIntervalSec=, TriggerLimitBurst=

Configures a limit on how often this socket unit may be activated within a specific time interval. The TriggerLimitIntervalSec= may be used to configure the length of the time interval in the usual time units "us", "ms", "s", "min", "h", … and defaults to 2s (See systemd.time(7) for details on the various time units understood). The TriggerLimitBurst= setting takes a positive integer value and specifies the number of permitted activations per time interval, and defaults to 200 for Accept=yes sockets (thus by default permitting 200 activations per 2s), and 20 otherwise (20 activations per 2s). Set either to 0 to disable any form of trigger rate limiting. If the limit is hit, the socket unit is placed into a failure mode, and will not be connectible anymore until restarted. Note that this limit is enforced before the service activation is enqueued.

[lahwaacz]

@travier That is very different mechanism than restarting. TriggerLimitIntervalSec=0 prevents the socket from entering failed state only due to hitting a trigger limit, but not due to e.g. out-of-memory conditions.

[travier]

Good point. This only works for the DoS case.

[YHNdnzj]

Try to change the dependency type to Upholds= (UpheldBy=)? While it's not customizable based on specific error, the use case of Restart=always should be covered greatly.

[travier]

Looks like https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Upholds= could indeed be a workaround but it's not great as it require another "fake" unit.

[lahwaacz]

If it works, the "fake" unit can be even sockets.target.

[YHNdnzj]

Looks like https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Upholds= could indeed be workaround but it's not great as it require another "fake" unit.

Why is something like that required? You can specify UpheldBy= and enable the socket as usual, no?

[travier]

OK, so something like below could potentially do it (needs testing) for the sshd.socket unit (or any other socket unit):

[Install]
UpheldBy=sockets.target

$ systemctl reenable sshd.socket

[YHNdnzj]

OK, so something like below could potentially do it (needs testing) for the sshd.socket unit (or any other socket unit):

[Unit]
UpheldBy=sockets.target

Just like WantedBy=, UpheldBy= is an [Install] setting. You should simply replace WantedBy with UpheldBy and reenable the socket.

[travier]

Good point, updated.

[travier]

For reference, #29159 should help with this.

[dustinlagoy]

I had an issue with a socket failing on boot, due to a network resource not yet being available, that could be worked around easily if this issue were resolved. I tried adding the socket to the Upholds= setting of sockets.target. This worked but doesn't solve the issue because the socket would restart quickly and hit a restart limit before the network resource came online. Setting the StartLimitIntervalSec to 0 for the socket doesn't help much because there is apparently a hardcoded check for 16 restarts within a 10 second interval (this is Unit.auto_start_stop_ratelimit currently set at

systemd/src/core/unit.c

Line 137 in f660c7f

u->auto_start_stop_ratelimit = (const RateLimit) {

).

For what it is worth the error due to the rate limit is:

Unit needs to be started because active unit sockets.target upholds it, but not starting since we tried this too often recently.

I worked around it with another service to monitor and restart the socket but it would be nice if systemd would handle this.

@travier I'm not sure #29159 would help here.

[travier]

@dustinlagoy This is not the same issue. If your socket depends on a specific network resource being up then you should order it after that, not bypass/workaround the retry logic in systemd.

[dustinlagoy]

Yes you are correct in my case that is a better solution. Perhaps for other cases there may be some need to work around the hardcoded rate limit though I don't know what they would be.