"Error: can only create exec sessions on running containers: container state improper" when running kind create cluster

[wheelerlaw]

What happened:

Command and output:

$ kind create cluster -v 100
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "kind" ...
DEBUG: podman/images.go:58] Image: kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✓ Preparing nodes 📦  
 ✗ Writing configuration 📜 
ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "podman exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 255
Command Output: Error: can only create exec sessions on running containers: container state improper
Stack Trace: 
sigs.k8s.io/kind/pkg/errors.WithStack
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/errors/errors.go:59
sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/exec/local.go:124
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman.(*nodeCmd).Run
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/cluster/internal/providers/podman/node.go:146
sigs.k8s.io/kind/pkg/exec.OutputLines
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/exec/helpers.go:81
sigs.k8s.io/kind/pkg/cluster/nodeutils.KubeVersion
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/cluster/nodeutils/util.go:35
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.getKubeadmConfig
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/cluster/internal/create/actions/config/config.go:208
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.(*Action).Execute.func1.1
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/cluster/internal/create/actions/config/config.go:90
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
        /home/wlaw/go/pkg/mod/sigs.k8s.io/kind@v0.11.1/pkg/errors/concurrent.go:30
runtime.goexit
        /usr/lib/golang/src/runtime/asm_amd64.s:1371

What you expected to happen:

A cluster to be created

How to reproduce it (as minimally and precisely as possible):

I installed kind using the following command:

GO111MODULE="on" go get sigs.k8s.io/kind@v0.11.1

Anything else we need to know?:

Environment:

• kind version: (use kind version): kind v0.11.1 go1.16.6 linux/amd64
• Kubernetes version: (use kubectl version): v1.21.4
• Docker version: (use docker info): podman 3.3.1
• OS (e.g. from /etc/os-release): Fedora 34

[BenTheElder]

if you run kind create cluster --retain (flag to prevent cleanup) and then share kind export logs we can look at what caused the node container to exit early. this is typically an issue with the host environment: https://kind.sigs.k8s.io/docs/user/known-issues/

[aojea]

Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/

are you running with podman rootless?

https://kind.sigs.k8s.io/docs/user/rootless/

[wheelerlaw]

Yes, I am running rootless. I followed the docs to add Delegate=yes to the systemd config and it fixed my issue, thanks.

[aojea]

awesome, Thanks