k3s-killall.sh does not stop containers

[std4453]

Environmental Info:
K3s Version:
k3s version v1.23.4+k3s1 (43b1cb4)
go version go1.17.5

Node(s) CPU architecture, OS, and Version:
Linux home 5.15.7-arch1-1 #1 SMP PREEMPT Wed, 08 Dec 2021 14:33:16 +0000 x86_64 GNU/Linux

Cluster Configuration:
1 node for development

Describe the bug:
Running k3s-killall.sh does not kill the containers.

Steps To Reproduce:

• Installed K3s with curl -sfL https://get.k3s.io | sh -
• Start some pods
• Run k3s-killall.sh

Expected behavior:
All containerd processes should be gone.

Actual behavior:
They're still there.

Additional context / logs:
On my system, the containerd processes are like:

root     1149329  0.0  0.0 708952  6656 ?        Sl   11:16   0:00 containerd-shim -namespace k8s.io -workdir /var/lib/rancher/k3s/agent/containerd/io.containerd.runtime.v1.linux/k8s.io/236cbef7cabe9346c9ee753db698d5c52951d03ac98c7d52cc38c171d64098ec -address /run/k3s/containerd/containerd.sock -containerd-binary /var/lib/rancher/k3s/data/4012316506613ee8c3cffc1e5b5eca706270685d33585804b257e93ea98d1917/bin/containerd

By looking at k3s-killall.sh, I can see:

k3s/install.sh

Lines 649 to 651 in 7710bce

	getshims() {
	ps -e -o pid= -o args= | sed -e 's/^ *//; s/\s\s*/\t/;' | grep -w 'k3s/data/[^/]*/bin/containerd-shim' | cut -f1
	}

Looks like it searches for processes containing k3s/data/[^/]*/bin/containerd-shim, where it finds none, and thus containers are not stopped.

I modified k3s-killall.sh to search for k3s/data/[^/]*/bin/containerd instead and it works without problems.

More context: In order to use nvidia GPU in cluster, I modified /var/lib/rancher/k3s/agent/etc/containerd/config.toml to be:

[plugins.cri.containerd.runtimes.runc]
  # ---- changed from 'io.containerd.runc.v2' for GPU support
  runtime_type = "io.containerd.runtime.v1.linux"

# ---- added for GPU support
[plugins.linux]
  runtime = "nvidia-container-runtime"

Anyway, I think k3s-killall.sh could just perhaps check for both .../containerd-shim as well as .../containerd to find all the container processes to kill.

[brandond]

K3s-killall.sh only kills containers run by the embedded containerd and runc. It does not attempt to match processes from alternative container runtimes, as there are many of these, and we are not interested in anticipating all the possible ones someone might use. Even if we did, we could potentially end up killing things that don't belong to us.

Also, note you don't need to modify the containerd config.toml to use the nvidia runtime. See the conversation at #4391 (comment)

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.