systemd-shutdown hangs on containerd-shim when k3s-agent running

[sourcedelica]

Environmental Info:
K3s Version: k3s version v1.18.6+k3s1 (6f56fa1)

Node(s) CPU architecture, OS, and Version: x86_64 Ubuntu 20.04.1
Linux nuc-linux3 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 1 master 2 workers

Describe the bug:
When shutting down or rebooting the node, the shutdown hangs for approximately 90 seconds. The console message is

systemd-shutdown: waiting for process: containerd-shim

When researching the problem I landed on this issue: ddev/ddev#2538 (comment) where they said when they uninstalled k3s the problem went away. I disabled and stopped k3s-agent.service and rebooted and the problem also went away for me.

I also tried re-enabling and starting k3s-agent.service and removing the docker.io package and running apt autoremove to remove containerd, runc, etc. but it still hangs on reboot at the same place.

[sourcedelica]

Following containerd/containerd#386 (comment) I changed the service configuration for k3s.agent and k3s-agent.service to KillMode=Mixed and that fixed the problem. This is in the standard Docker configuration.

However, I also found #1965 where it looks like this behavior is as intended. Is there a way to allow for upgrading k3s without disrupting workloads but at the same time not hang shutdowns/reboots for 90s?

[sourcedelica]

I was thinking one way to do it is to use KillMode=mixed or KillMode=control-group by default for k3s{-agent}.service and when doing an upgrade, add a drop-in in /run/systemd/system/k3s{-agent}.service.d that temporarily sets KillMode=process before stopping the service, then removes the drop-in after the upgrade.

[dontlaugh]

systemd has an explicit pre-shutdown hook, so perhaps you could invoke special logic with that. See:

/usr/lib/systemd/system/shutdown.target.wants

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[unixfox]

Bump still relevant

[senpaiSubby]

bump same issue

[andrewchen5678]

notice the issue with raspberry pi with display on

[jraby]

I faced this issue yesterday and ended up with the following solution.

/etc/systemd/system/cgroup-kill-on-shutdown@.service :

[Unit]
Description=Kill cgroup procs on shutdown for %i
DefaultDependencies=false
Before=shutdown.target umount.target
[Service]
# Instanced units are not part of system.slice for some reason
# without this, the service isn't started at shutdown
Slice=system.slice
ExecStart=/bin/bash -c 'pids=$(cat /sys/fs/cgroup/unified/system.slice/%i/cgroup.procs); echo $pids | xargs -r kill;'                                                                                                                                                                      
ExecStart=/bin/sleep 5                                                                                                                                                                                                                                                                     
ExecStart=/bin/bash -c 'pids=$(cat /sys/fs/cgroup/unified/system.slice/%i/cgroup.procs); echo $pids | xargs -r kill -9;'
Type=oneshot
[Install]
WantedBy=shutdown.target

Enable the "service" for k3s-agent.service (will also work for k3s on the master ):

sudo systemctl enable cgroup-kill-on-shutdown@k3s-agent.service.service

# or, on the master:  sudo systemctl enable cgroup-kill-on-shutdown@k3s.service.service

I've written a long winding explanation here but in brief, what happens is that since killmode=process is used, all the container processes end up staying alive when k3s is brought down. Which is a good thing ™️

However, during shutdown, systemd will signal all remaining processes and wait for DefaultTimeoutStopSec for them to die.
This is always 90s during the last shutdown phase with systemd v245.
It is a bug in systemd v245 shipped with ubuntu 20.04 and was fixed in september 2020

What I used to do was to set DefaultTimeoutStopSec=5s in /etc/systemd/system.conf and it worked fine, but on ubuntu 20.04 it doesn't.

Since there's little chance this fix will make it back into 20.04, the above "service" will perform round of SIGTERM, wait 5s, then proceed with SIGKILL to finish k3s's process cleanup during shutdown.
The sleep can be tweaked to suit your services need (something matching terminationGracePeriod perhaps)

Hope it helps.

[sourcedelica]

Awesome research!

[miraculixx]

@jraby your solution helped me to resolve the issue, however I ended up using the k3s-killall.sh according to the k3s docs . With this there is no shutdown delay on my system.

Caution - this may not be what you want

The killall script cleans up containers, K3s directories, and networking components while also removing the iptables chain with all the associated rules. The cluster data will not be deleted.

I'm using this /etc/systemd/system/cgroup-kill-on-shutdown@.service

# source https://github.com/k3s-io/k3s/issues/2400#issuecomment-1013798094
# $ sudo systemctl enable cgroup-kill-on-shutdown@k3s-agent.service.service
[Unit]
Description=Kill cgroup procs on shutdown for %i
DefaultDependencies=false
Before=shutdown.target umount.target
[Service]
# Instanced units are not part of system.slice for some reason
# without this, the service isn't started at shutdown
Slice=system.slice
ExecStart=/bin/bash -c "/usr/local/bin/k3s-killall.sh"
Type=oneshot
[Install]
WantedBy=shutdown.target

This is on

Linux Mint 20.2
5.4.0-91-generic

[horihel]

the same problem exists on rke2 (no surprise, given its roots are in k3s)

[brandond]

Yes, this is by design. Stopping the K3s (or RKE2) service does not stop running containers. This is to allow for nondisruptive upgrades of the main K3s/RKE2 components by simply replacing the binary and restarting the service.

[horihel]

would you accept a feature request to add a systemd unit like #2400 (comment) which only triggers on shutdown?
This would both allow the intended behaviour of k3s/rke2 (seamless updates/restarts) and allow for a shutdown/reboot that's even quicker than RKE1.

here's my non-instanced version of that (for rke2):

[Unit]
Description=Kill containerd-shims on shutdown
DefaultDependencies=false
Before=shutdown.target umount.target

[Service]
ExecStart=/bin/bash -c "/usr/local/bin/rke2-killall.sh"
Type=oneshot

[Install]
WantedBy=shutdown.target