[liquorix kernel] error when running kind create cluster: failed to create cluster: failed to init node with kubeadm

[bdutta]

What happened:
Got an error when running kind create cluster
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0429 15:20:52.212263 141 initconfiguration.go:201] loading configuration from "/kind/kubeadm.conf"

What you expected to happen:
Expected successful cluster creation.

How to reproduce it (as minimally and precisely as possible):
Not sure if this'd help reproduce in all environments, but here is what I did:

1. Ensured latest version of docker-ce (from configured repos) is installed on my Ubuntu MATE 20.04. This happens to be Docker 20.10.6
2. Ensure latest version of go-lang is installed on this system. This happens to be 1.13.8.
3. Installed kind using recommended method for Linux in general:
   curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.10.0/kind-linux-amd64 chmod +x ./kind mv ./kind $HOME/go/src/

Anything else we need to know?:
Full output available here - https://pastebin.com/4ZReTRCC

Environment:
Linux on bare-metal (Intel i5-8400 w/ 16GB DDR4)

• kind version: (use kind version): kind v0.10.0 go1.13.8 linux/amd64
• Kubernetes version: (use kubectl version): 1.21.0
• Docker version: (use docker info): 20.10.6
• OS (e.g. from /etc/os-release): Ubuntu MATE 20.4 w/ liquorix kernel (5.11.0-13.1-liquorix-amd64)

[bdutta]

I think I found the issue based on this (which I'd missed during first scan):

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

AFAIU, the liquorix kernel that I've used for music-production, doesn't support cgroups. I suppose, changing to Ubuntu stock kernel would solve the issue.

[BenTheElder]

Lack of cgroups will definitely cause this

[Arnavion]

0.10.0 can't deploy v1.21 clusters, with our without cgroups v2 support. See #2189 , where BenTheElder suggested waiting for 0.11.0 / testing with master.

[bdutta]

The k8s version is determined by kubectl version, right ? If so, can downgrading kubectl to an older version, s.a. v1.19 or even v1.18 help ? That is, assuming that I switch to the Ubuntu generic or lowlatency (stock) kernel (that has requisite cgroups support) ?

[Arnavion]

kubectl has nothing to do with it. You control the node image used via the --image parameter of kind create cluster. If you give it a 1.19 image it'll deploy a 1.19 cluster.

[BenTheElder]

@Arnavion I suspect only their client version is 1.21.0 given that they did not get a cluster up so kubectl would not be able to report the server version (though we could tell definitively in the future if the full output of kubectl version were included).

This likely has nothing to do with 1.21.0 support since the 1.21 clients will be fine with 1.20 clusters (kind v0.10 default).

It is probably the custom kernel. https://techpatterns.com/forums/about2796.html

There's nothing we can do about that, you'll need to install a kernel that supports cgroups (like a mainline kernel) or use a different tool.

/remove-kind bug
/kind support

[Arnavion]

Oh right, sorry. I forgot kind has a default cluster version since I always use it with an explicit one.

[BenTheElder]

#2852 (comment) we've had a similar issue filed recently, seems it's an issue with the kernel as suggested here in #2224 (comment) by bdutta