kind cluster creation fails

[qcloutier-sonrai]

Actual Behavior

Attempting to create a K8s cluster using kind fails on the preparing nodes step.

Steps to Reproduce

$ kind create cluster

Result

Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 
 ✗ Preparing nodes 📦  
Deleted nodes: ["kind-control-plane"]
ERROR: failed to create cluster: command "docker run --name kind-control-plane --hostname kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=kind --net kind --restart=on-failure:1 --init=false --cgroupns=private --publish=127.0.0.1:53569:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72" failed with error: exit status 125
Command Output: 8069484d7cb04377ca7b3d19cc7ee39c2bfa00c8f253c993346ecee10509b7b9
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/7), flags: 0xe, data: openrc: invalid argument: unknown.

Expected Behavior

The cluster is created without error

Additional Information

Works fine on both Docker Desktop and Podman Desktop.

Rancher Desktop Version

1.9.1

Rancher Desktop K8s Version

N/A

Which container engine are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

macOS Ventura 13.4.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

None

Windows User Only

No response

[ericpromislow]

I can reproduce this on both an intel-mac and ubuntu, so I'm assuming something is needed in the Lima VM.

Reformatting the above session so it's easier to read:

$ rdctl shell
$ docker run --name kind-control-plane --hostname kind-control-plane \
--label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined \
--security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var \
--volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER \
--detach --tty --label io.x-k8s.kind.cluster=kind --net kind --restart=on-failure:1 --init=false \
--cgroupns=private --publish=127.0.0.1:64033:6443/TCP \
-e KUBECONFIG=/etc/kubernetes/admin.conf \
kindest/node:v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba7216f087f1ce8bf7049c141c43a4e18e98660b4d76d701a24b9d3fbae54a7132ee

docker: Error response from daemon: failed to create shim task: 
        OCI runtime create failed: runc create failed: unable to start container process: 
        error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup":
        mount cgroup:/sys/fs/cgroup/openrc (via /proc/self/fd/7), flags: 0xe, data: openrc: invalid argument: unknown.

Without quotes it's hard to parse that final data: openrc: invalid argument: unknown. bit

[ericpromislow]

Some other output while in rdctl shell sudo su -:

# cd /sys/fs/cgroup
lima-rancher-desktop:/sys/fs/cgroup# ls
blkio       cpuacct     devices     hugetlb     net_cls     openrc      pids
cpu         cpuset      freezer     memory      net_prio    perf_event  unified
# ls openrc
acpid                  cgroup.sane_behavior   lima-guestagent        sshd
cgroup.clone_children  crond                  notify_on_release      tasks
cgroup.procs           docker                 release_agent

[ericpromislow]

From kubernetes-sigs/kind#3277 (comment) it looks like this is a limitation caused by our VM being based on alpine and not one that uses cgroup v2 (kubernetes-sigs/kind#3277 (comment))

[jandubois]

We have an open issue to investigate switching to cgroupv2: #4141.

[BenTheElder]

On cgroupv2 cgroupns=private is the default, but it's settable on v1.

docker/podman both change the default based on v1/v2 unless the user overrides it, but there's no reason it can't be enabled on v1.

The problem seems to be on these alpine hosts (with the unusual init?) cgroupns=private containers fail (the change in kind v0.20.0), this client flag was added in docker 20.10.0 released 2020-12-08.

[ck3mp3r]

I can confirm the same error with KinD running against Colima on an arm based mac...

[jandubois]

I've tentatively added this to the 1.11 milestone to investigate if we can make it work with Alpine/OpenRC.

[donovat]

The same issue is also effecting buildx build issues, when you create a new buildx instance. Using the default instance is fine, so its the new container created as part of the docker buildx create command.

[Symbianx]

Any news on this? Seems like it was removed from the 1.12 milestone and no fix in sight. Currently, neither rancher desktop nor colima support kind.

I really like Rancher Desktop and want to recommend it in our org, without kind support it's hard to do so.

[jandubois]

I'm sorry, but I'm not aware of any way to get kind working on Alpine Linux, even when you get past the cgroup v2 issues.

The best I can suggest at this time is to try out k3d, which is similar to kind but uses k3s internally instead of kubeadm. I believe it offers pretty much the same functionality.

[BenTheElder]

If someone has time to debug this further kind maintainers can review incoming patches or debugging information if it turns out to be a bug in kind but I'm not realistically able to focus on this specific host anytime soon. For the moment you'll need a different pairing of tools, there are some suggestions in the linked kind issue above.