K3s in Docker with systemd (cgroupfs v1)

[apcheamitru]

I am trying to run K3s 1.24 in a Docker container with systemd, but haven't had any luck. I believe this worked in earlier K3s versions until #5462, which changed the default cgroup driver from "cgroupfs" to "systemd" when systemd has been detected.

My host is configured to use cgroupfs v1 (via systemd.unified_cgroup_hierarchy=0) and my Docker image is based on debian:11. I'm spawning the container with: docker run -td --privileged -v /sys/fs/cgroup:/sys/fs/cgroup --name k3s <image> and systemd is PID 1. The only option I'm passing to K3s is snapshotter=native.

I see that the K3s node never becomes ready:

root@d9160e850009:/# kubectl get no -o wide
NAME           STATUS     ROLES    AGE   VERSION         INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION          CONTAINER-RUNTIME
d9160e850009   NotReady   <none>   9s    v1.24.11+k3s1   172.17.0.2    <none>        Debian GNU/Linux 11 (bullseye)   4.18.0-448.el8.x86_64   containerd://1.6.15-k3s1

When I describe the node, I see:

  Ready            False   Mon, 10 Apr 2023 19:52:26 +0000   Mon, 10 Apr 2023 19:51:52 +0000   KubeletNotReady              [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]

And in the K3s journald logs I see this error:

Apr 10 19:51:52 d9160e850009 k3s[79]: E0410 19:51:52.594710      79 kubelet.go:1378] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist"

I'm not really sure what to make of this, so I tried with the "cgroupfs" cgroup driver instead. Here's my K3s config:

root@9fa788a0d84b:/# cat /etc/rancher/k3s/config.yaml
---
snapshotter: native
kubelet-arg:
  - --cgroup-driver="cgroupfs"

This time I see a "Ready" node, but Pods are stuck in ContainerCreating:

root@9fa788a0d84b:/# kubectl get no -o wide
NAME           STATUS   ROLES                  AGE   VERSION         INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION          CONTAINER-RUNTIME
9fa788a0d84b   Ready    control-plane,master   14s   v1.24.11+k3s1   172.17.0.2    <none>        Debian GNU/Linux 11 (bullseye)   4.18.0-448.el8.x86_64   containerd://1.6.15-k3s1

root@9fa788a0d84b:/# kubectl get po -A
NAMESPACE     NAME                                      READY   STATUS              RESTARTS   AGE
kube-system   local-path-provisioner-687d6d7765-lv77z   0/1     ContainerCreating   0          14s
kube-system   helm-install-traefik-jjjh4                0/1     ContainerCreating   0          15s
kube-system   coredns-7b5bbc6644-bm794                  0/1     ContainerCreating   0          14s
kube-system   metrics-server-667586758d-m56dm           0/1     ContainerCreating   0          14s
kube-system   helm-install-traefik-crd-kfbfw            0/1     ContainerCreating   0          15s

Describing the Pods shows:

  Warning  FailedCreatePodSandBox  3s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/kubepods/besteffort/pode134b605-2254-4715-b548-db5895999c41/d2af8bc542eb5389b0f6007b2daf17039b3fb4413f84551158f2bf4e35b3ee73" instead: unknown

I believe this is because containerd is still using the "systemd" cgroup driver:

root@9fa788a0d84b:/# grep Cgroup /var/lib/rancher/k3s/agent/etc/containerd/config.toml
  SystemdCgroup = true

But changing that value to "false" and deleting Pods didn't seem to change the behavior I'm seeing 🤔

Has anyone had luck getting K3s to run in Docker with systemd? If so, could you please share your configuration? I'm trying to integrate with a Docker-based test framework and would really like to see this working!

[brandond]

My host is configured to use cgroupfs v1 (via systemd.unified_cgroup_hierarchy=0) and my Docker image is based on debian:11. I'm spawning the container with: docker -td --privileged -v /sys/fs/cgroup:/sys/fs/cgroup --name k3s and systemd is PID 1.

I'm confused, you're running a custom Docker image with systemd running in docker that is in turn starting K3s as a systemd unit within that Docker container? That's not something we've ever tested or really support. Why not just run the rancher/k3s image directly? That should work regardless of whether you're using cgroup v1 or v2.

[apcheamitru]

That's correct. The full background is that I have Puppet managing the installation and configuration of K3s along with other node configurations and am using the Puppetlabs Litmus test framework with the Docker provisioner for acceptance testing. It's not perfect, but good enough in most cases. I was looking to do some simple assertions, like validating registries.yaml, for example.

I just found that I can template the containerd config with config.toml.tmpl, so that's enough to force everything to use "cgroupfs". I'm not sure if I understand the repercussions of doing so, but it gets Pods to run and might be good enough for testing 🤔

Hmm... now that I think about it though... if that approach is reasonable, maybe the cgroupfs driver shouldn't default to "systemd" when Docker + systemd is detected? 🤔🤷‍♂️

[brandond]

K3s ensures that the kubelet and containerd cgroup driver are both set to the same thing, whether that's cgroupfs or systemd. You're not overriding the kubelet's cgroup driver via kubelet args, are you?

Assuming you're not doing that, I guess my question would be why the containerd cgroup driver is broken when containerd is running in docker with cgroupv1. I'm not sure how exactly I would go about detecting "systemd in docker in cgroupv1" or if that's even a common enough case that we need to handle it.

[apcheamitru]

Ah, yes, I was overriding the kubelet's cgroup driver via kubelet args.

I had initially tried without overriding the cgroup driver (which detected "systemd") but was met with failed to initialize top level QOS containers: root container [kubepods] doesn't exist. I tried "cgroupfs" to work around that.

Is there a way to specify "cgroupfs" or "systemd" without kubelet args? Or a better way to approach this with the "systemd" cgroup driver?

[brandond]

Are you actually running k3s as a systemd unit, using the default systemd unit that we provide? I personally will occasionally run systemd in docker and then install k3s into that environment and it works fine, but I think systemd requires some extra configuration to work properly. The image I use for this is at

• https://github.com/rancher/systemd-node
• https://hub.docker.com/layers/rancher/systemd-node/v0.0.4/images/sha256-c4d5fb5df795e889cd9b55acce1956482e3f43b8a45ac09b3162ac580ba8af90?context=explore

[apcheamitru]

Yes, I am running K3s as a systemd unit using the stock systemd unit. It could be something on my host? I was able to reproduce the same behavior using rancher/systemd-node:v0.0.4 (see below).

Spin up Docker container:

$ docker run -td --privileged --name k3s rancher/systemd-node:v0.0.4
6ea5b7d2821aecf71604a9ce6d7270dd49375112d17f10a05ccf5ab7b008d616

Install and run K3s:

$ docker exec -it k3s bash
6ea5b7d2821a:/ # export INSTALL_K3S_SELINUX_WARN=true
6ea5b7d2821a:/ # curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.26.3+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.26.3+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.26.3+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[WARN]  Failed to find the k3s-selinux policy, please install:
    zypper install -y container-selinux
    zypper install -y https://rpm.rancher.io/k3s/stable/common/microos/noarch/k3s-selinux-1.2-2.sle.noarch.rpm

[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

Same error:

6ea5b7d2821a:/ # journalctl -fu k3s
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: I0411 14:55:21.329393     347 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: I0411 14:55:21.329566     347 policy_none.go:49] "None policy: Start"
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: I0411 14:55:21.339642     347 memory_manager.go:169] "Starting memorymanager" policy="None"
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: I0411 14:55:21.340120     347 state_mem.go:35] "Initializing new in-memory state store"
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: I0411 14:55:21.340440     347 state_mem.go:75] "Updated machine memory state"
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: I0411 14:55:21.350306     347 kubelet_node_status.go:70] "Attempting to register node" node="6ea5b7d2821a"
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: E0411 14:55:21.360406     347 kubelet.go:2137] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Apr 11 14:55:21 6ea5b7d2821a k3s[347]: E0411 14:55:21.380110     347 kubelet.go:1466] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: root container [kubepods] doesn't exist"
Apr 11 14:55:21 6ea5b7d2821a systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 14:55:21 6ea5b7d2821a systemd[1]: k3s.service: Failed with result 'exit-code'.
^C

[brandond]

This is a bit of a hack, but can you try doing echo INVOCATION_ID= >> /etc/default/k3s.env? That will fake k3s into thinking it's on an old version of systemd that cannot be used as the cgroup driver.

[apcheamitru]

That worked! Thank you very much @brandond!

(Note: I used /etc/systemd/system/k3s.service.env instead of /etc/default/k3s.env.)

[brandond]

Be careful, as that file will be overwritten by the install script, which is why I recommended a different one.