rootless: regression between May 20-June 8: failed to "CreatePodSandbox":: ...: conmon cgroup system.slice invalid for cgroupfs

[AkihiroSuda]

Description

rootless-containers/usernetes@3637546

Rootless mode got broken between May 20 and June 8 (ad83d2a...ed409ae)

Steps to reproduce the issue:

1. Checkout rootless-containers/usernetes@3637546
2. Set CRIO_COMMIT in Dockerfile
3. DOCKER_BUILDKIT=1 docker build -t rootlesscontainers/usernetes .
4. ./hack/smoketest-docker.sh u7s-test-crio rootlesscontainers/usernetes --cri=crio

Describe the results you received:
ad83d2a (May 20) works, ed409ae (June 8) does not.

https://github.com/rootless-containers/usernetes/runs/752788072

  Jun 09 06:54:28 40718833cf6a kubelet-crio.sh[282]: E0609 06:54:28.178025      70 remote_runtime.go:113] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = conmon cgroup system.slice invalid for cgroupfs
  Jun 09 06:54:28 40718833cf6a kubelet-crio.sh[282]: E0609 06:54:28.178066      70 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "hello_default(bf4d3ec8-e479-403f-8ba6-32f9d61f5006)" failed: rpc error: code = Unknown desc = conmon cgroup system.slice invalid for cgroupfs
  Jun 09 06:54:28 40718833cf6a kubelet-crio.sh[282]: E0609 06:54:28.178078      70 kuberuntime_manager.go:724] createPodSandbox for pod "hello_default(bf4d3ec8-e479-403f-8ba6-32f9d61f5006)" failed: rpc error: code = Unknown desc = conmon cgroup system.slice invalid for cgroupfs
  Jun 09 06:54:28 40718833cf6a kubelet-crio.sh[282]: E0609 06:54:28.178108      70 pod_workers.go:191] Error syncing pod bf4d3ec8-e479-403f-8ba6-32f9d61f5006 ("hello_default(bf4d3ec8-e479-403f-8ba6-32f9d61f5006)"), skipping: failed to "CreatePodSandbox" for "hello_default(bf4d3ec8-e479-403f-8ba6-32f9d61f5006)" with CreatePodSandboxError: "CreatePodSandbox for pod \"hello_default(bf4d3ec8-e479-403f-8ba6-32f9d61f5006)\" failed: rpc error: code = Unknown desc = conmon cgroup system.slice invalid for cgroupfs"

Describe the results you expected:

It should work

[AkihiroSuda]

Suspicious commit: ac96653

[haircommander]

if you're using cgroupfs, you need to change conmon_cgroup to pod, or some other cgroup path. it was mentioned in the release notes section in #3810

[haircommander]

Suspicious commit: ac96653

your suspicion is correct

[mrunalp]

@AkihiroSuda @haircommander We could add a rootless flag to the cgroup manager if needed.

[haircommander]

We could also just detect if we're rootless and not fatally error if so. Though, I am pretty sure @AkihiroSuda's configuration would fail as root as well, so I'm curious if it would be fixed by fixing the conmon_cgroup

[AkihiroSuda]

Thanks for the answer