CreatePodSandbox errors in RPI 4 cluster

[serversonic]

I have three RPI 4s in a cluster using the 18.04.3 image from berryboot. Microk8s setup is as per the documentation. Running microk8s enable dns appears to complete, but kubectl get all --all-namespaces then shows

kube-system pod/coredns-588fd544bf-clsrh 0/1 ContainerCreating 0 29m
and
journalctl -f -u snap.microk8s.daemon-kubelet shows

Jun 06 19:55:42 cluster01 microk8s.daemon-kubelet[2239]: E0606 19:55:42.784516 2239 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/work upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/fs lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown Jun 06 19:55:42 cluster01 microk8s.daemon-kubelet[2239]: E0606 19:55:42.784747 2239 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "coredns-588fd544bf-clsrh_kube-system(8349ceb4-2a99-46b1-8922-9c6f79584233)" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/work upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/fs lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown Jun 06 19:55:42 cluster01 microk8s.daemon-kubelet[2239]: E0606 19:55:42.784835 2239 kuberuntime_manager.go:727] createPodSandbox for pod "coredns-588fd544bf-clsrh_kube-system(8349ceb4-2a99-46b1-8922-9c6f79584233)" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/work upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/fs lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown Jun 06 19:55:42 cluster01 microk8s.daemon-kubelet[2239]: E0606 19:55:42.785105 2239 pod_workers.go:191] Error syncing pod 8349ceb4-2a99-46b1-8922-9c6f79584233 ("coredns-588fd544bf-clsrh_kube-system(8349ceb4-2a99-46b1-8922-9c6f79584233)"), skipping: failed to "CreatePodSandbox" for "coredns-588fd544bf-clsrh_kube-system(8349ceb4-2a99-46b1-8922-9c6f79584233)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-588fd544bf-clsrh_kube-system(8349ceb4-2a99-46b1-8922-9c6f79584233)\" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/work upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1601/fs lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown"

The rpc error would seem to be the problem, which seems to occur in other k8s and docker configurations.

inspection-report-20200606_200834.tar.gz

We appreciate your feedback. Thank you for using microk8s.

[balchua]

Hi @serversonic i see this in the `dmesg

[10720.209668] overlayfs: filesystem on '/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1525/fs' not supported as upperdir

Are you using zfs?

[serversonic]

I'd seen that and started looking at it last night df -Thshows the filesystem as overlay:

df -Th
Filesystem Type Size Used Avail Use% Mounted on
dev devtmpfs 759M 0 759M 0% /dev
none overlay 458G 13G 422G 3% /
tmpfs tmpfs 888M 0 888M 0% /dev/shm
tmpfs tmpfs 888M 832K 887M 1% /run

So the error makes sense. Any fixes or workarounds welcome but I should be able to reinstall as a proper filesystem if necessary.

[balchua]

Im not sure if containerd supports overlay over overlay. Can you try making it a proper filesystem?

[serversonic]

I think that's the issue, it didn't occur to me that the OS would be installed as anything other than a valid filesystem. I'll rebuild the disks and report back.

[balchua]

Thanks @serversonic ! Let us know how it goes.

[serversonic]

Got the cluster rebuilt with ext4 filesystems using the Ubuntu arm64 RPI image. There was another problem with enabling cgroup memory as it isn't enabled in RPI 4, which may be worth adding as a note somewhere (basically add cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1 to /boot/firmware/nobtcmd.txt). System pods are now enabled and I can access the dashboard remotely.

[ktsakalozos]

To change the snapshotter you will need to edit /var/snap/microk8s/current/args/containerd-template.toml and stop/start microk8s. You may have luck with some other snapshotter.

[ktsakalozos]

There was another problem with enabling cgroup memory as it isn't enabled in RPI 4

We have this in the docs [1], but we may want to detect and display an error message in the microk8s inspect script [2].

[1] https://microk8s.io/docs/install-alternatives#heading--arm
[2] https://github.com/ubuntu/microk8s/blob/master/scripts/inspect.sh

[serversonic]

I took an inspect dump to try and identify the problem myself:
inspection-report-20200607_190944.tar.gz

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.