Kubernetes example unstable on newer OS

[afbjorklund]

For some reason, the k8s.yaml example is unstable when running on the newer distribution versions.

Tried with both the default ubuntu impish and fedora 35, but get strange timeouts and crash loops etc.

The only difference I can think of, is the cgroups v2 ?

Running under Ubuntu 20.04 LTS has no issues...

[afbjorklund]

I think that this is an upstream issue (we don't do anything special at all, outside the READMEs)

Would like to get some help with testing, or if there is some extra configuration that k3s is doing ?

https://github.com/lima-vm/lima/blob/master/examples/k3s.yaml
https://github.com/k3s-io/k3s/blob/master/install.sh

---

The example uses focal for now, but eventually it would be nice to run the same distribution release.

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

[AkihiroSuda]

Could you provide more information of the issue? Do you see any error in the logs of kubelet/containerd/others ?

[afbjorklund]

I think it needs to be run multiple times, so far I am mostly seeing internal timeouts and CrashLoopBackoffs.

I was expecting the unloaded cluster to (eventually) come up every time, but I guess it needs debug logs...

Also why I wanted help with troubleshooting it.

If it is only me, or happens everywhere ?

--- a/examples/k8s.yaml
+++ b/examples/k8s.yaml
@@ -11,11 +11,10 @@
 
 # This example requires Lima v0.7.0 or later.
 images:
-  # Image is set to focal (20.04 LTS) for long-term stability
   # Hint: run `limactl prune` to invalidate the "current" cache
-  - location: "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
+  - location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.img"
     arch: "x86_64"
-  - location: "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img"
+  - location: "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-arm64.img"
     arch: "aarch64"
 mounts:
   - location: "~"

[afbjorklund]

Just thought these looked a bit suspicous: (from kubelet)

manager.go:159] Cannot detect current cgroup on cgroup v2
dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
server.go:687] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
container_manager_linux.go:280] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]

NAMESPACE     NAME                               READY   STATUS             RESTARTS        AGE
kube-system   coredns-78fcd69978-6nrd5           1/1     Running            1 (3m31s ago)   5m
kube-system   coredns-78fcd69978-xtwhz           1/1     Running            1 (3m32s ago)   5m
kube-system   etcd-lima-k8s                      1/1     Running            2 (3m3s ago)    5m27s
kube-system   kube-apiserver-lima-k8s            1/1     Running            2 (59s ago)     5m32s
kube-system   kube-controller-manager-lima-k8s   1/1     Running            3 (78s ago)     5m27s
kube-system   kube-flannel-ds-8zncc              1/1     Running            0               5m1s
kube-system   kube-proxy-mwfvb                   0/1     CrashLoopBackOff   3 (22s ago)     5m1s
kube-system   kube-scheduler-lima-k8s            0/1     CrashLoopBackOff   3 (41s ago)     5m53s

Eventually, it stops.

The connection to the server 192.168.5.15:6443 was refused - did you specify the right host or port?

[afbjorklund]

This is the normal output: (with ubuntu 20.04 LTS)

NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-js4nr           1/1     Running   0          14s
kube-system   coredns-78fcd69978-kfhb5           1/1     Running   0          14s
kube-system   etcd-lima-k8s                      1/1     Running   0          22s
kube-system   kube-apiserver-lima-k8s            1/1     Running   0          21s
kube-system   kube-controller-manager-lima-k8s   1/1     Running   0          22s
kube-system   kube-flannel-ds-qdsqp              1/1     Running   0          14s
kube-system   kube-proxy-dkxtk                   1/1     Running   0          14s
kube-system   kube-scheduler-lima-k8s            1/1     Running   0          28s

[AkihiroSuda]

CrashLoopBackOff

kubectl --namespace=kube-system describe pod POD may show more detailed error

[afbjorklund]

It says something about Pod sandbox changed, it will be killed and re-created.