Cilium broken on Amazon Linux 2 and RHEL8

[rifelpet]

/kind bug

copied from #12141 (comment)

https://testgrid.k8s.io/kops-grid#kops-grid-cilium-rhel8-k22-containerd

kubelet logs report:

E0916 00:19:04.290207 18754 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/host-path/23431797-5c17-4404-b61b-33d2e8ec3d54-cilium-cgroup podName:23431797-5c17-4404-b61b-33d2e8ec3d54 nodeName:}" failed. No retries permitted until 2021-09-16 00:19:36.290185341 +0000 UTC m=+151.386939478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cilium-cgroup" (UniqueName: "kubernetes.io/host-path/23431797-5c17-4404-b61b-33d2e8ec3d54-cilium-cgroup") pod "cilium-8rdpr" (UID: "23431797-5c17-4404-b61b-33d2e8ec3d54") : mkdir /sys/fs/cgroup/unified: read-only file system

We could mount cgroup2 explicitly somewhere else as well, like we used to with bpffs. Or we could drop support for not-that-modern-distros.

In #11696 we deprecated CentOS 8 but not RHEL 8. AL2 is still the most modern version of Amazon Linux so I dont think we should be deprecating it. I think it would be best if we mount cgroup2 elsewhere.

[rifelpet]

Also we may want to consider this blocking 1.22 given the change was introduced in 1.22 and is breaking clusters for supported distros.

[olemarkus]

I agree. I'll revive the mount functionality and have it mount cgroup2 on the default cilium location (/run/cilium/cgroupv2)

[olemarkus]

RHEL8's kernel is too old for cilium. But #12431 should take care of AL2 5.10 kernel

[rifelpet]

Here are the cilium docs mentioning the kernel version requirements for some of the more advanced features. That page also mentions an overall requirement of 4.9 and that RHEL8 is supported. Is our manifest using the advanced features by default?

[olemarkus]

I am not sure I'd say this is an "advanced feature" anymore as it has been the default in cilium for quite some time, and for kOps as well. But "Kubernetes Without kube-proxy" is enabled by default (on new clusters) as of kOps 1.19.

[rifelpet]

In that case maybe we add a 1.22 release note mentioning that cilium + RHEL8 is not supported. We can skip those grid jobs too (and add AL2)

[johngmyers]

@rifelpet is the current release note sufficient?