CI failing for cgroup2+docker+rootless

[AkihiroSuda]

CI is recently failing for cgroup2+docker+rootless.

CI isn't failing for cgroup2+docker+rootful and cgroup2+podman+rootless.

Also the failure is not reproducible on my local Vagrant.

https://github.com/kubernetes-sigs/kind/runs/3255789909

kind-logs-cgroup2-docker-rootless-1102670304.zip:kind-control-plane/kubelet.log

-- Journal begins at Thu 2021-08-05 19:38:30 UTC, ends at Thu 2021-08-05 19:40:32 UTC. --
Aug 05 19:38:31 kind-control-plane systemd[1]: Condition check resulted in kubelet: The Kubernetes Node Agent being skipped.
Aug 05 19:38:36 kind-control-plane systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Aug 05 19:38:36 kind-control-plane sh[217]: ERROR: this script needs /sys/fs/cgroup/cgroup.procs to be empty (for writing the top-level cgroup.subtree_control)
Aug 05 19:38:36 kind-control-plane systemd[1]: kubelet.service: Control process exited, code=exited, status=1/FAILURE
Aug 05 19:38:36 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.
Aug 05 19:38:36 kind-control-plane systemd[1]: Failed to start kubelet: The Kubernetes Node Agent.
Aug 05 19:38:37 kind-control-plane systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Aug 05 19:38:37 kind-control-plane systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Aug 05 19:38:37 kind-control-plane systemd[1]: Starting kubelet: The Kubernetes Node Agent...
Aug 05 19:38:37 kind-control-plane sh[220]: ERROR: this script needs /sys/fs/cgroup/cgroup.procs to be empty (for writing the top-level cgroup.subtree_control)
Aug 05 19:38:37 kind-control-plane systemd[1]: kubelet.service: Control process exited, code=exited, status=1/FAILURE
Aug 05 19:38:37 kind-control-plane systemd[1]: kubelet.service: Failed with result 'exit-code'.
Aug 05 19:38:37 kind-control-plane systemd[1]: Failed to start kubelet: The Kubernetes Node Agent.
...

[AkihiroSuda]

CI was passing for #2407 (Docker 20.10.8, k 1.21.2), but began to fail for #2406 (Docker 20.10.8, k 1.22.0).

Regression in the base image?

[aojea]

Regression in the base image?

or in kubernetes 1.22?

[AkihiroSuda]

Weird, the issue is locally reproducible with "$HELPER" kind create cluster -v7 --wait 10m --retain but not with the same kind command in vagrant ssh shell.

Could be related to some tty stuff.
(EDIT: adding -tt to the ssh command in $HELPER seemed to work for me locally, but doesn't work on CI)