k8s.yaml example does not work after upgrade to Ubuntu 22.04

[kingdonb]

Description

I discovered lima yesterday and I was glad to find that k8s.yaml provides everything I need to run a disposable test instance of Kubeadm that can successfully connect to my synology-csi through their open source driver, (other solutions like k3d, kind all did not succeed in my trials).

However this morning I saw an upgrade was made from 20.04 to 22.04 ubuntu, and now etcd seems to restart for some unknown reason every 3-5 minutes and it results in cluster services dropping out and all sorts of chaos inside the cluster.

It's easy enough to revert to the previous commit and I have done that, and everything seems to be fine again. But I thought I should report this, since I see a lot of things were upgraded at once, it is possible that nobody is aware, (or maybe this is an issue with my machine, but it seems not.)

I am using an M1 Macbook and the only changes I made to the VM config were to add:

cpus: 5
memory: 8GiB

Just from glancing at the htop statistics in the VM before things go to crap, it does not look like we're running out of memory or anything else jumping out at me as the reason for this fault.

[afbjorklund]

There was similar issues with 21.10 before

• Kubernetes example unstable on newer OS #378

Similar problems were seen with Fedora 35

https://github.com/afbjorklund/fedora-lima/blob/main/fedora-kubernetes.yaml

[afbjorklund]

Could try with k8s 1.24 and flannel 0.17, to see if it works better ?

But I think that it is related to the cgroup driver, since it changed

[afbjorklund]

There seems to be something more than just changing the cgroups version, though.

NAMESPACE     NAME                               READY   STATUS             RESTARTS        AGE     IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-64897985d-4rj6s            1/1     Running            2 (96s ago)     5m      10.244.0.6     lima-k8s   <none>           <none>
kube-system   coredns-64897985d-nvmxc            1/1     Running            2 (35s ago)     5m      10.244.0.7     lima-k8s   <none>           <none>
kube-system   etcd-lima-k8s                      1/1     Running            3 (2m2s ago)    5m22s   192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-apiserver-lima-k8s            1/1     Running            1 (5m32s ago)   5m48s   192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-controller-manager-lima-k8s   1/1     Running            3 (104s ago)    5m50s   192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-flannel-ds-stcx6              1/1     Running            0               5m      192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-proxy-klqrr                   0/1     CrashLoopBackOff   3 (24s ago)     5m      192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-scheduler-lima-k8s            1/1     Running            3 (85s ago)     5m49s   192.168.5.15   lima-k8s   <none>           <none>

But it is a start.

EDIT: Forgot that we already tried this, last time. And "systemd" is the default, since 1.22

[afbjorklund]

Can't really see what is going on, except for etcd shutting down and then stopping the rest

{"level":"info","ts":"2022-04-25T17:02:00.461Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}

[core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

[kingdonb]

This looks like my same issue, (I mentioned that etcd was restarting, but everything else restarts like you show – I assumed because etcd was unstable it caused leader election and other baseline configuration things to fail these ways in other svcs)

[afbjorklund]

Tried changing containerd installation, from nerdctl-full (1.6.2) to ubuntu (1.5.9) but it didn't seem to help

containerd:
  system: false
  user: false

apt-get install -y containerd runc

systemctl enable --now containerd

[afbjorklund]

But changing from containerd-shim-runc-v2 to containerd-shim-runc-v1 seems to improve the situation

systemd.unified_cgroup_hierarchy=0 (grub)

cgroupfs-mount (apt)

runtime_type = "io.containerd.runc.v1" (containerd)

[afbjorklund]

Changing containerd version, from 0.8.7 to 1.1.1, didn't help either (adjusted cni config, accordingly)

    mv -i /opt/cni/bin /opt/cni/bin.old
    ln -s /usr/local/libexec/cni /opt/cni/bin

[afbjorklund]

Here is a command to get a list of all the pods, including the ones that are currently crashed:

limactl shell k8s sudo crictl ps -a

CONTAINER           IMAGE               CREATED              STATE               NAME                      ATTEMPT             POD ID
1c2e91cac8852       df7b72818ad2e       21 seconds ago       Running             kube-controller-manager   2                   9ee33f24de3f2
acb1e30c5ad89       595f327f224a4       21 seconds ago       Running             kube-scheduler            2                   9efdbb876f14f
40fb9bb3ed1ff       25f8c7f3da61c       24 seconds ago       Running             etcd                      3                   a8519e9bcb072
35e4516aee653       a4ca41631cc7a       37 seconds ago       Running             coredns                   1                   d2e6c63830d06
66b8455d74a9b       4c03754524064       44 seconds ago       Running             kube-proxy                3                   4be512a3bc488
f8ae7bbb5e134       a4ca41631cc7a       About a minute ago   Exited              coredns                   0                   7098da5416e8e
a045334e18fb7       a4ca41631cc7a       About a minute ago   Running             coredns                   0                   3fbf271ac6ab8
07e4c374b112e       25f8c7f3da61c       2 minutes ago        Exited              etcd                      2                   7456e823125e9
1f7348e8bdd1a       4c03754524064       2 minutes ago        Exited              kube-proxy                2                   9b061f25d3f05
4f6d3e5fe866e       8fa62c12256df       2 minutes ago        Running             kube-apiserver            1                   b95d9e45688c3
492927b1b46be       595f327f224a4       2 minutes ago        Exited              kube-scheduler            1                   9efdbb876f14f
69c4e32e005fe       df7b72818ad2e       2 minutes ago        Exited              kube-controller-manager   1                   9ee33f24de3f2
41d4625c6ed2b       8fa62c12256df       3 minutes ago        Exited              kube-apiserver            0                   18b67efe1a30e

Since kubectl might not always reply, if kube-apiserver or etcd is down.

limactl shell k8s sudo kubectl get pods -A -o wide

NAMESPACE     NAME                               READY   STATUS    RESTARTS        AGE     IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-64897985d-2pkxh            1/1     Running   1 (71s ago)     4m55s   10.88.0.6      lima-k8s   <none>           <none>
kube-system   coredns-64897985d-8p6lp            1/1     Running   2 (84s ago)     4m56s   10.88.0.5      lima-k8s   <none>           <none>
kube-system   etcd-lima-k8s                      1/1     Running   3 (2m48s ago)   5m1s    192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-apiserver-lima-k8s            1/1     Running   1 (4m44s ago)   4m41s   192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-controller-manager-lima-k8s   1/1     Running   3 (52s ago)     5m2s    192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-proxy-jn5l9                   1/1     Running   4 (79s ago)     4m56s   192.168.5.15   lima-k8s   <none>           <none>
kube-system   kube-scheduler-lima-k8s            1/1     Running   3 (36s ago)     5m2s    192.168.5.15   lima-k8s   <none>           <none>

[afbjorklund]

@AkihiroSuda : can we downgrade the k8s.yaml to Ubuntu 20.04, similar to the nomad.yaml example ?

At least until the support in Kubernetes is out of beta, or until we know why it isn't working out-of-the-box.

https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2254-cgroup-v2

I have not tried using 1.22 or 1.24, just 1.23: https://dl.k8s.io/release/stable-1.txt (currently v1.23.6)

[AkihiroSuda]

Downgrading SGTM for now 😞
Do we want to downgrade k3s.yaml too?

[afbjorklund]

I'm not sure if this is related or not:

Cannot detect current cgroup on cgroup v2

Apparently it checks /sys/fs/cgroup ?

cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

[afbjorklund]

Do we want to downgrade k3s.yaml too?

The rancher cluster does look much happier:

NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE   IP          NODE       NOMINATED NODE   READINESS GATES
kube-system   local-path-provisioner-84bb864455-9smlx   1/1     Running     0          42m   10.42.0.2   lima-k3s   <none>           <none>
kube-system   coredns-96cc4f57d-qnt7v                   1/1     Running     0          42m   10.42.0.4   lima-k3s   <none>           <none>
kube-system   helm-install-traefik-crd--1-89k62         0/1     Completed   0          42m   10.42.0.5   lima-k3s   <none>           <none>
kube-system   metrics-server-ff9dbcb6c-9bkhk            1/1     Running     0          42m   10.42.0.6   lima-k3s   <none>           <none>
kube-system   helm-install-traefik--1-6xdj8             0/1     Completed   2          42m   10.42.0.3   lima-k3s   <none>           <none>
kube-system   svclb-traefik-lsbpj                       2/2     Running     0          41m   10.42.0.7   lima-k3s   <none>           <none>
kube-system   traefik-56c4b88c4b-fchvb                  1/1     Running     0          41m   10.42.0.8   lima-k3s   <none>           <none>

Even if running with containerd-shim-runc-v2

NAME       STATUS   ROLES                  AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION      CONTAINER-RUNTIME
lima-k3s   Ready    control-plane,master   44m   v1.22.7+k3s1   192.168.5.15   <none>        Ubuntu 22.04 LTS   5.15.0-25-generic   containerd://1.5.9-k3s1

So this seems to be local to k8s.yaml/kubeadm.