bump default node image

[aojea]

No description provided.

[aojea]

/assign @BenTheElder

[aojea]

I have the feeling that podman rootless broke with the new podman 3.4.0 release

[aojea]

confirmed, new podman 3.4.0 rootless doesn't work with this kind image

Oct 15 11:10:07 kind-control-plane kubelet[151]: E1015 11:10:07.781642 151 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:58: preparing rootfs caused: permission denied: unknown" pod="kube-system/kube-scheduler-kind-control-plane"

@AkihiroSuda can it be related to some of the overlay mounts?

[aojea]

failed job https://github.com/kubernetes-sigs/kind/pull/2502/checks?check_run_id=3905205991
comparing without the node image bump
#2504

[AkihiroSuda]

https://github.com/kubernetes-sigs/kind/suites/4057989849/artifacts/103222385

kind-control-plane/serial.log:

WARN: UserNS: SELinux might be Enforcing. If you see an error related to overlayfs, try setting KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs .

podman-info.txt: kernel: 5.11.12-300.fc34.x86_64

---

Workarounds:

• Modify the entrypoint to use fuse-overlayfs by default, when SELinux seems to be enabled.
  This is the easiest fix, but decreases performance.

kind/images/base/files/usr/local/bin/entrypoint

Lines 82 to 88 in 220bd84

	# Detect whether SELinux is Enforcing (or Permitted) by grepping /proc/self/attr/current .
	# Note that we cannot use `getenforce` command here because /sys/fs/selinux is typically not mounted for containers.
	if grep -q "_t:" "/proc/self/attr/current"; then
	# When the kernel is before v5.13 and SELinux is enforced, fuse-overlayfs might be safer, so we print a warning (but not an error).
	# https://github.com/torvalds/linux/commit/7fa2e79a6bb924fa4b2de5766dab31f0f47b5ab6
	echo "WARN: UserNS: SELinux might be Enforcing. If you see an error related to overlayfs, try setting \`KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs\` ." >&2
	fi

• Just run the CI with KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs and call it a day

• Just run the CI without SELinux and call it a day.

• Upgrade the kernel to 5.13 or later, which contains fixes for overlayfs on SELinux.
  Fedora 35 will be released soon with kernel 5.14.

[aojea]

something we've modified in the entrypoint broke podman :/
#2504 (comment)

EDIT
Sorry, I didn't see the message

[aojea]

• Upgrade the kernel to 5.13 or later, which contains fixes for overlayfs on SELinux.
  Fedora 35 will be released soon with kernel 5.14.

if is a kernel problem I'm in favor on using the environment variable, let's see if it works

[aojea]

Just run the CI with KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER=fuse-overlayfs and call it a day

it doesn't work 👀

[AkihiroSuda]

hack/ci/vagrant-helper.sh has to be modified to propagate $KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER

[aojea]

trying with selinux, I've realized we are already doing that for docker ...

[aojea]

ok, disabling selinux works and is consistent with current docker rootless CI,
I think that is fair until distros pick newer kernels with the fixed mentioned by Akihiro
@BenTheElder @AkihiroSuda do you agree?

[AkihiroSuda]

Looks good, thanks

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AkihiroSuda, aojea, BenTheElder

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [BenTheElder,aojea]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BenTheElder]

SGTM, we should probably re-enable it later.