Failed to Create Cluster on M1

[RohanMishra97]

What happened:
Creating a kind cluster for the first time, I installed it on my Macbook M1 using brew install kind followed by kind create cluster --retain I got the following error

 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 137

kind export logs
Archive.zip
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
Getting the following logs in the container.

INFO: remounting /sys read-only
INFO: making mounts shared
INFO: detected cgroup v1
INFO: fix cgroup mounts for all subsystems
INFO: clearing and regenerating /etc/machine-id
Initializing machine ID from random generator.
INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

Environment:

• kind version: (use kind version): kind v0.11.1 go1.16.4 darwin/arm64
• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/arm64"}
• Docker version: (use docker info): Server: Containers: 1 Running: 0 Paused: 0 Stopped: 1 Images: 3 Server Version: 20.10.8 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux Default Runtime: runc Init Binary: docker-init containerd version: e25210fe30a0a703442421b0f60afac609f950a3 runc version: v1.0.1-0-g4144b63 init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.47-linuxkit Operating System: Docker Desktop OSType: linux Architecture: aarch64 CPUs: 4 Total Memory: 7.765GiB Name: docker-desktop ID: BEFJ:DBR6:F7HN:UK6X:H5GF:3DYT:RDFX:RSC5:CHMJ:CJ6T:JXP6:NJLU Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
• OS (e.g. from /etc/os-release): sw_vers ProductName: macOS ProductVersion: 11.5.2 BuildVersion: 20G95

[RohanMishra97]

As a workaround, having a Dockerfile

FROM --platform=arm64 kindest/node:v1.21.1
RUN arch

building it
docker build -t tempkind .
and using that image
kind create cluster --image tempkind does the job.
Was the amd64 image getting pulled? I was running the commands on arm64 terminal.

[BenTheElder]

I'm only breifly around today, probably won't be able to follow up myself for a while but no, the image is being pulled by a mult-arch manifest list digest (not amd64 specific).

Probably what happened is docker desktop switched to iptables backed by the nftables kernel module, the latest kind images have updated logic for this. when you use your custom image you are pulling kindest/node:v1.21.1 without pinning the digest to a previous known version and getting the latest image.

that's my best guess anyhow, you could dig deeper and find out what digests are being used and what the image differences are (e.g. I recommend wagoodman/dive for inspecting images). I could also tell you if I knew which were used.

for v0.11.1 the default is

kind/pkg/apis/config/defaults/image.go

Line 21 in 6a69d86

const Image = "kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6"

[BenTheElder]

This part is the problem.

INFO: setting iptables to detected mode: legacy
iptables-save v1.8.7 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

#2289 may be the fix (Already merged). I would be interested to see if using kind built from the latest sources works (the default should contain this fix).

[BenTheElder]

cc @munnerz 👀

[rushminatorr]

I am having similar errors as specified by BenTheElder with regard to iptables error.
running Kind on macOS, no docker installed but using podman.

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: podman run error: command "podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume a6984b1aed9fb8dace52bae4fa022031f57336b823c88414888444e34b9a4cbf:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:56363:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 126
Command Output: Error: error configuring network namespace for container 0f72ac51c27d1460f30c94f868f03b976aaf946e433b1c782ecbd23c5127c00e: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to list chains: running [/usr/sbin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

[aojea]

I am having similar errors as specified by BenTheElder with regard to iptables error. running Kind on macOS, no docker installed but using podman.

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: podman run error: command "podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume a6984b1aed9fb8dace52bae4fa022031f57336b823c88414888444e34b9a4cbf:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:56363:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 126
Command Output: Error: error configuring network namespace for container 0f72ac51c27d1460f30c94f868f03b976aaf946e433b1c782ecbd23c5127c00e: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to list chains: running [/usr/sbin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

I assume you are using podman machine, that doesn't work with kind AFAIK #2213 (comment)

[rushminatorr]

@aojea yes thank you for directing me to the link - you are right!

[BenTheElder]

Podman machine has different open issues, and kind is known to work on M1 mac by many users, something clearly went wrong in the initial post here, but we don't seem to have a reproducer and I don't think there's anything more to do with that initial post.

I'm going to close this now so it stops splitting topics to podman machine etc., those should be seperate issues please, and please fill out the bug template with all the details so we know enough about your environment to discuss / diagnose.

#2445 for podman machine on mac

[vlaguduva]

I am having similar errors as specified by BenTheElder with regard to iptables error. running Kind on macOS, no docker installed but using podman.

❯ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: podman run error: command "podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume a6984b1aed9fb8dace52bae4fa022031f57336b823c88414888444e34b9a4cbf:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:56363:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 126
Command Output: Error: error configuring network namespace for container 0f72ac51c27d1460f30c94f868f03b976aaf946e433b1c782ecbd23c5127c00e: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to list chains: running [/usr/sbin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

You can overcome this problem if you init the podman machine as rootful.