Error validating CNI config on create cluster - kind with rootless podman Ubuntu 22.04

[rpete3]

What happened: Error validating CNI config on create cluster

kind create cluster --retain
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✗ Preparing nodes 📦
ERROR: failed to create cluster: command "podman run --name kind-control-plane --hostname kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 2ad4eed591a54d9fa9620331defcc4265f4e5c39db8ac51275ee9c366e656b5c:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --cgroupns=private --device /dev/fuse --publish=127.0.0.1:33571:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf docker.io/kindest/node@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72" failed with error: exit status 127
Command Output: time="2023-06-20T12:20:18-04:00" level=warning msg="Error validating CNI config file /home/petersr/.config/cni/net.d/kind.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]"
time="2023-06-20T12:20:18-04:00" level=warning msg="Error validating CNI config file /home/petersr/.config/cni/net.d/kind.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]"
time="2023-06-20T12:20:18-04:00" level=error msg="error loading cached network config: network "kind" not found in CNI cache"
time="2023-06-20T12:20:18-04:00" level=warning msg="falling back to loading from existing plugins on disk"
time="2023-06-20T12:20:18-04:00" level=warning msg="Error validating CNI config file /home/petersr/.config/cni/net.d/kind.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]"
time="2023-06-20T12:20:18-04:00" level=error msg="Error tearing down partially created network namespace for container dabe6e4f972348a84b23fbbdaf5caf9c51c396b8291332b6bc6e1eb5592ce96a: CNI network "kind" not found"
Error: error configuring network namespace for container dabe6e4f972348a84b23fbbdaf5caf9c51c396b8291332b6bc6e1eb5592ce96a: CNI network "kind" not found
petersr@U-xxxxxxx:~$ kind export logs
enabling experimental podman provider
Exporting logs for cluster "kind" to:
/tmp/2220968499
ERROR: [command "podman exec --privileged kind-control-plane sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 255, [command "podman exec --privileged kind-control-plane journalctl --no-pager -u containerd.service" failed with error: exit status 255, command "podman exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 255, command "podman exec --privileged kind-control-plane journalctl --no-pager" failed with error: exit status 255, command "podman exec --privileged kind-control-plane crictl images" failed with error: exit status 255, command "podman exec --privileged kind-control-plane journalctl --no-pager -u kubelet.service" failed with error: exit status 255]]

What you expected to happen:
Create cluster.

How to reproduce it (as minimally and precisely as possible):
install via binary and run on latest ubuntu 22.04 lts
Anything else we need to know?:

Environment:

• kind version: (use kind version): v0.19.0 and v0.20.0

• Runtime info: (use docker info or podman info): host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:

  • memory
  • pids
    cgroupManager: systemd
    cgroupVersion: v2
    conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
    cpus: 2
    distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
    eventLogger: journald
    hostname: U-2xxxxxxxx
    idMappings:
    gidmap:
    • container_id: 0
      host_id: 943200513
      size: 1
    • container_id: 1
      host_id: 100000
      size: 65536
      uidmap:
    • container_id: 0
      host_id: 943201112
      size: 1
    • container_id: 1
      host_id: 100000
      size: 65536
      kernel: 5.19.0-1027-aws
      linkmode: dynamic
      logDriver: journald
      memFree: 2116771840
      memTotal: 8216158208
      ociRuntime:
      name: crun
      package: 'crun: /usr/bin/crun'
      path: /usr/bin/crun
      version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
      os: linux
      remoteSocket:
      exists: true
      path: /run/user/943201112/podman/podman.sock
      security:
      apparmorEnabled: false
      capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
      rootless: true
      seccompEnabled: true
      seccompProfilePath: /usr/share/containers/seccomp.json
      selinuxEnabled: false
      serviceIsRemote: false
      slirp4netns:
      executable: /usr/bin/slirp4netns
      package: 'slirp4netns: /usr/bin/slirp4netns'
      version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
      swapFree: 15971995648
      swapTotal: 16432312320
      uptime: 12h 44m 36.98s (Approximately 0.50 days)
      plugins:
      log:
  • k8s-file
  • none
  • journald
    network:
  • bridge
  • macvlan
    volume:
  • local
    registries:
    docker.io:
    Blocked: false
    Insecure: false
    Location: ntr.nwie.net/docker.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: docker.io
    quay.io:
    Blocked: false
    Insecure: false
    Location: ntr.nwie.net/quay.io
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: quay.io
    search:
  • docker.io
    store:
    configFile: /home/petersr/.config/containers/storage.conf
    containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
    graphDriverName: overlay
    graphOptions: {}
    graphRoot: /home/petersr/.local/share/containers/storage
    graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
    imageStore:
    number: 2
    runRoot: /run/user/943201112/containers
    volumePath: /home/petersr/.local/share/containers/storage/volumes
    version:
    APIVersion: 3.4.4
    Built: 0
    BuiltTime: Wed Dec 31 19:00:00 1969
    GitCommit: ""
    GoVersion: go1.18.1
    OsArch: linux/amd64
    Version: 3.4.4

• OS (e.g. from /etc/os-release):PRETTY_NAME="Ubuntu 22.04.2 LTS"
  2 │ NAME="Ubuntu"
  3 │ VERSION_ID="22.04"
  4 │ VERSION="22.04.2 LTS (Jammy Jellyfish)"
  5 │ VERSION_CODENAME=jammy
  6 │ ID=ubuntu
  7 │ ID_LIKE=debian
  8 │ HOME_URL="https://www.ubuntu.com/"
  9 │ SUPPORT_URL="https://help.ubuntu.com/"
  10 │ BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  11 │ PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/pri
  │ vacy-policy"
  12 │ UBUNTU_CODENAME=jammy

• Kubernetes version: (use kubectl version): lient Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.15", GitCommit:"2c67202dc0bb96a7a837cbfb8d72e1f34dfc2808", GitTreeState:"clean", BuildDate:"2023-06-15T02:16:07Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.4

• Any proxies or other special environment settings?:

[jeremyfritzen]

Hi!

I have the same issue since today.

WARN[0000] Error validating CNI config file /home/podman/.config/cni/net.d/torrenting_default.conflist: [plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"]

[aojea]

that seems a podman problem, please try with a newer version or report it in podman containers/podman#13396

kind support for podman is experimental

/kind support
/area podman
/remove-kind bug

[k8s-ci-robot]

@aojea: The label(s) area/podman cannot be applied, because the repository doesn't have them.

In response to this:

that seems a podman problem, please try with a newer version or report it in podman containers/podman#13396

kind support for podman is experimental

/kind support
/area podman
/remove-kind bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

Yeah, this is not a kind bug, we don't create any CNI config for podman.

[rpete3]

It’s a Podman issue. Upgrading to a more current version has resolved the issue. That said, the podman CNI config error did not occur until I tried to create a kind cluster. Sent from Proton Mail for iOS

…

On Wed, Aug 2, 2023 at 2:22 PM, Benjamin Elder ***@***.***(mailto:On Wed, Aug 2, 2023 at 2:22 PM, Benjamin Elder <<a href=)> wrote: Yeah, this is not a kind bug, we don't create any CNI config for podman. — Reply to this email directly, [view it on GitHub](#3280 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AWY5CZTOT6PK2GWMBP64ZZTXTKLERANCNFSM6AAAAAAZNRKYUA). You are receiving this because you authored the thread.Message ID: ***@***.***>

[BenTheElder]

It’s a Podman issue. Upgrading to a more current version has resolved the issue. That said, the podman CNI config error did not occur until I tried to create a kind cluster.

kind requests a different network from the default which is triggering the issue, but it's not responsible for the CNI config, that part is internal to podman.

[BenTheElder]

kind asks podman to create a network, but it doesn't ask for anything particularly unusual and the CNI config is generated internal to podman.