Single node RKE2 stopping cluster

[whgibbo]

Hi,
Noticed that when I stop the rke2-server services using:

systemctl stop rke2-agent.service

That the pods are not shutdown.. When checking the status I see :

rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/lib/systemd/system/rke2-server.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Fri 2025-07-25 14:36:43 BST; 2s ago
   Duration: 4min 47.522s
       Docs: https://github.com/rancher/rke2#readme
    Process: 2957930 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 2957932 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 2957933 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 2957934 ExecStart=/usr/bin/rke2 server (code=exited, status=1/FAILURE)
    Process: 2967838 ExecStopPost=/bin/sh -c systemd-cgls /system.slice/rke2-server.service | grep -Eo '[0-9]+ (containerd|kubelet)' | awk '{print $1}' | xargs -r kill (code=exited, status=0/SUCCESS)
   Main PID: 2957934 (code=exited, status=1/FAILURE)
      Tasks: 162
     Memory: 98.2M
        CPU: 1min 24.129s
     CGroup: /system.slice/rke2-server.service
             ├─2958599 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1205f508223701a08cb849dcdbaee5d51c6bd2af610bcec24ad5b7aaa4be015f -address /run/k3>
             ├─2958620 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id d7cc0bc2f787a6118b2785fa6060d807f5b55b8a094eaae565224ba03e865c06 -address /run/k3>
             ├─2958770 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id a163959bad0cc51806a182b7b3001387ce351a9a94e67e38d5bb9efa516838af -address /run/k3>
             ├─2958861 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id f62318733f1add9a99deab711937dbd65821207753bf2b0870a7b50736711b79 -address /run/k3>
             ├─2958872 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9dae6720b26cf38cb207eb31d0aa62adf3bf255bb3e98bb90cc47da99d372a6c -address /run/k3>
             ├─2958910 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id b8aff760d7f54b3b84be634ff17afaae2b1732d6358dbb7cf2930c4edc129523 -address /run/k3>
             ├─2959387 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id ed47c2e0f77bd41f7b8847adbdb7c2590a43167e8571ff9057886da74bec425e -address /run/k3>
             ├─2960824 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id dde6a91bc93bbf0b6fb025ef13e6046ed250ae36361cac7fbb5fd7b5852ac8b8 -address /run/k3>
             ├─2960947 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4abc0d2fb04dbdf2fce06072aa2dfba207d8b74e9ed4077e603bd0249f793094 -address /run/k3>
             ├─2960960 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1a2bcfcf5fdfb7ee349321cc0ba7ffe02ea1ef41b3659926d8a23aa302dedcb6 -address /run/k3>
             ├─2961039 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id 36a393607e08ab1275a480084edf350adb58c856f99e1ec192cdb6b56053e8c8 -address /run/k3>
             ├─2961406 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id df42658853a7f34a56a9d30c3dbfb6c3d76fe25fd01aaff780c9cf6205283671 -address /run/k3>
             └─2962166 /var/lib/rancher/rke2/data/v1.32.6-rke2r1-95e28c150059/bin/containerd-shim-runc-v2 -namespace k8s.io -id c7f4b461558c64dd7f777928be995a30f2e19514972feb5ec75f2b26983122cc -address /run/k3>

Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2958910 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2959387 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2960824 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2960947 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2960960 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2961039 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2961406 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Unit process 2962166 (containerd-shim) remains running after unit stopped.
Jul 25 14:36:43 localhost.localdomain systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jul 25 14:36:43 localhost.localdomain systemd[1]: rke2-server.service: Consumed 1min 24.129s CPU time.

Or should I be using rke2-killall.sh

[brandond]

That is correct. Rke2 uses containerd as it's container runtime. Stopping containerd and/or the kubelet does not stop running containers.

If you want to drain the node first, see the upstream docs on graceful node shutdown, or you can just run the killall script.

[whgibbo]

@brandond do you have a link..

As when I try

kubectl drain $(hostname) --ignore-daemonsets --delete-emptydir-data

I noticed that the due to the helmcontroller manifests it starts re-deploying them.

Also is there any plan to incorporate this within the rk2-server shutdown command?

[brandond]

do you have a link..

Google is your friend - did you search for "kubernetes graceful node shutdown"?

• https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#graceful-node-shutdown
• Graceful Node shutdown Single Node Cluster k3s-io/k3s#4319

I noticed that the due to the helmcontroller manifests it starts re-deploying them.

By "them" are you referring to the helm job pods? That is normal. There are no plans to alter helm controller behavior in this respect - all it does is make sure that the Jobs for HelmCharts exist and are in sync with the chart resources. Syncing Pods for the Job is handled by the kube-controller-manager, and not something that we handle ourselves.