kind create cluster failed on Fedora 36 with podman

[janvlug]

What happened:
I use Fedora 36 with podman. When I (as a normal user) run: kind -v 1 create cluster

I get:

enabling experimental podman provider
ERROR: failed to create cluster: running kind with rootless provider requires setting systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Stack Trace: 
sigs.k8s.io/kind/pkg/errors.New
	sigs.k8s.io/kind/pkg/errors/errors.go:28
sigs.k8s.io/kind/pkg/cluster/internal/create.validateProvider
	sigs.k8s.io/kind/pkg/cluster/internal/create/create.go:253
sigs.k8s.io/kind/pkg/cluster/internal/create.Cluster
	sigs.k8s.io/kind/pkg/cluster/internal/create/create.go:70
sigs.k8s.io/kind/pkg/cluster.(*Provider).Create
	sigs.k8s.io/kind/pkg/cluster/provider.go:182
sigs.k8s.io/kind/pkg/cmd/kind/create/cluster.runE
	sigs.k8s.io/kind/pkg/cmd/kind/create/cluster/createcluster.go:80
sigs.k8s.io/kind/pkg/cmd/kind/create/cluster.NewCommand.func1
	sigs.k8s.io/kind/pkg/cmd/kind/create/cluster/createcluster.go:55
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.4.0/command.go:902
sigs.k8s.io/kind/cmd/kind/app.Run
	sigs.k8s.io/kind/cmd/kind/app/main.go:53
sigs.k8s.io/kind/cmd/kind/app.Main
	sigs.k8s.io/kind/cmd/kind/app/main.go:35
main.main
	sigs.k8s.io/kind/main.go:25
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1571

What you expected to happen:
A cluster to be created without error messages.

Anything else we need to know?:
When I run the command as root, it executes without issues.

$ cat /etc/systemd/system/user@.service.d/delegate.conf
[Service]
Delegate=yes

Environment:

• kind version: (use kind version): kind v0.14.0 go1.18.2 linux/amd64
• Kubernetes version: (use kubectl version):

$ kubectl version --output=yaml
clientVersion:
  buildDate: "2022-06-18T00:00:00Z"
  compiler: gc
  gitCommit: 3ddd0f45aa91e2f30c70734b175631bec5b5825a
  gitTreeState: archive
  gitVersion: v1.24.1
  goVersion: go1.18.3
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4

• Docker version: (use docker info):

$ docker version
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.3
Built:        Wed Jun 22 18:17:44 2022
OS/Arch:      linux/amd64

• OS (e.g. from /etc/os-release):

$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="36 (Workstation Edition)"

Note issue #1902 is very similar, but I thought it was better to open a new issue because that issue is closed.

This is the output of docker info: docker_info.txt

[aojea]

networkBackend: netavark

You have to install the cni plugins, there is already another issue about this, I will try to find it later

[janvlug]

another issue about this, I will try to find it later

Maybe you mean issue #2821, I tried dnf install containernetworking-plugins podman-plugins, but this did not solve the issue.

[stmcginnis]

Also, did you go through the steps in https://kind.sigs.k8s.io/docs/user/rootless/ ?

[janvlug]

did you go through the steps

Yes, I did. To me it looks now like a permission issue somehow, because running as root this works:

# kind create cluster

and then:

# kind load docker-image hello-world

But as a regular user kind create cluster gives an error.

[janvlug]

This is the # docker info ran as root:
docker_info_root.txt

There is an interesting difference between this output ran as root compared to the output when run as regular user in the context of cgroupControllers:

  host:
    arch: amd64
    buildahVersion: 1.26.1
    cgroupControllers:
+   - cpuset
+   - cpu
+   - io
    - memory
+   - hugetlb
    - pids
+   - misc

[janvlug]

Edit: I deleted this comment, because it was incorrect.

[BenTheElder]

Rootless podman should still be setting up the cgroups controllers, this seems like a podman bug or misconfiguration.

I don’t think we should be documenting groups podman packaging setup, this may be distro specific and using podman rootfully is something podman should cover.

[janvlug]

I noticed that my previous comment was incorrect.
For currently still unknown reasons to me, there are more cgroupControllers when I ssh into my own machine with ssh localhost. When I ssh into my own machine, I can, also as a regular user, create a cluster with kind create cluster, also without being in the dockerroot group, contrary to what I previously, incorrectly, stated.
See also this discussion.
Edit: See also this bug: containers/podman#13710
Edit2: See also this bug: #2684 (comment)

[caniko]

Rootless podman should still be setting up the cgroups controllers, this seems like a podman bug or misconfiguration.

I don’t think we should be documenting groups podman packaging setup, this may be distro specific and using podman rootfully is something podman should cover.

Maybe get some podman devs involved? They can provide a PR for the docs possibly?

[BenTheElder]

@caniko I mean that this is out of scope for our own docs, we're intermittently in contact with folks working on podman but I would not ask them to document this here.

We don't cover installing docker either.

Kubernetes has a policy of not duplicating third party documentation. It's a waste of energy to maintain in N+1 places. Docs regarding installing podman belong in podman's documentation.

[BenTheElder]

@janvlug that's interesting and strange behavior ... is it containers/podman#13710 (comment) ?

[BenTheElder]

We've merged some other podman fixes since this bug was opened, but I still suspect this is something like containers/podman#13710 (comment)