Error validating CNI config file

[cmurf]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. podman system reset

Describe the results you received:

WARN[0000] Error validating CNI config file /home/chris/.config/cni/net.d/87-podman.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]] 

Describe the results you expected:

No errors on reset. Or I guess it should automatically or prompt to remove stale files (?)

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 4.0.1

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.24.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: workstation
    version: "36"
  eventLogger: journald
  hostname: fovo.local
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.17.0-0.rc6.109.fc36.x86_64+debug
  linkmode: dynamic
  logDriver: journald
  memFree: 10419982336
  memTotal: 16365338624
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-2.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 6m 37.43s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/chris/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/chris/.local/share/containers/storage
  graphStatus:
    Build Version: Btrfs v5.16.2
    Library Version: "102"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/chris/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.1
  Built: 1645815878
  BuiltTime: Fri Feb 25 12:04:38 2022
  GitCommit: ""
  GoVersion: go1.18beta2
  OsArch: linux/amd64
  Version: 4.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.0.1-2.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

This is a clean install of Fedora-Workstation-Live-x86_64-36-20220220.n.0.iso, updated. But I am using /home from a previous Fedora 35 installation so it might have some stale podman 3 bits; nevertheless the reset should get rid of it. But I get the WARN each time the command is run.

[flouthoc]

@cmurf I think your distro is missing cni plugins or your path variable does have /usr/sbin where these plugins are usually present.

Could you share output of

which bridge

[cmurf]

$ which bridge
/usr/sbin/bridge

[flouthoc]

@cmurf Thanks for sharing, podman by default reads CNI plugin from these paths https://github.com/containers/common/blob/main/pkg/config/default.go#L83 . If you are using custom path you would have to update containers.conf https://github.com/containers/common/blob/main/docs/containers.conf.5.md#network-table

or you could copy all the binaries and move to any of the default paths.

[baude]

that very well could be the bridge binary from iproute2. try:

rpm -q containernetworking-plugins

and then rpm -ql containernetworking-plugins to see where they are installed.

[cmurf]

$ rpm -q containernetworking-plugins
package containernetworking-plugins is not installed

Seems like podman needs a recommends or requires?

[cmurf]

Downstream bug https://bugzilla.redhat.com/show_bug.cgi?id=2060200

[rhatdan]

Since this is a packaging issue and not related to upstream, I am closing.

[zhangguanzhang]

containernetworking-plugins is in the appstream

$ dnf config-manager --set-enabled AppStream
$ dnf install containernetworking-plugins
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:00:21 ago on Wed Apr 27 02:01:14 2022.
Dependencies resolved.
============================================================================================================================================================================================================================================
 Package                                                          Architecture                                Version                                                                  Repository                                      Size
============================================================================================================================================================================================================================================
Installing:
 containernetworking-plugins                                      x86_64                                      1.0.0-1.module_el8.5.0+890+6b136101                                      appstream                                       19 M
Enabling module streams:
 container-tools                                                                                              rhel8                                                                                                                        

Transaction Summary
============================================================================================================================================================================================================================================
Install  1 Package

Total download size: 19 M
Installed size: 62 M
Is this ok [y/N]: 

[marquesmateus93]

Thanks @zhangguanzhang .

In Fedora 36 Silverblue:
$ rpm-ostree install containernetworking-plugins

Then reboot.

[Verhoeckx]

In Fedora 36 Silverblue: $ rpm-ostree install containernetworking-plugins
Then reboot.

Perfect, that was the solution!
Has this been reported as a bug on the Fedora Silverblue issues list?

[acxz]

If anyone is coming from archlinux, the relevant package is cni-plugins.
i.e. pacman -S cni-plugins

[kristofer-lp8f]

Ran into same error when using podman-compose. Found an issue in Silverblue issue tracker fedora-silverblue/issue-tracker#246

[xxxserxxx]

@acxz Any idea why this isn't a dependency in the podman-compose package?

Why does it seem to be happening on multiple distributions (Redhat, Arch)? Did something change in podman? I've been using podman-compose on a container (without having cni-plugins installed) without issue on podman 4.1.0 until I upgraded to 4.1.1 (a minor revision bump) a couple of weeks ago.

[acxz]

Any idea why this isn't a dependency in the podman-compose package?

My guess is that CNI is more of a configuration thing, instead of a required dep. You can probably get more information about this decision by talking to the Arch maintainers for podman (@Foxboron, @dvzrv).

It makes sense for me that it should be listed as an optdepend for podman, but not really sure. It would probably be best for someone let the Arch maintainers know about this.

Did something change in podman?

I would assume that podman moved the cni code that was internal to the codebase and moved it outside, which caused such behavior.

[dvzrv]

AFAIK, netavark is supposed to be the "new thing" to use (both cni-plugins and netavark provide "container-network-stack", but in different versions - 1 and 2 respectively). On Arch Linux podman optionally depends on netavark, but also depends on containers-common which requires container-network-stack>1. Netavark is therefore made the explicit default by now.

My understanding so far was, that podman still works with both providers of container-network-stack. I guess the only thing we can do is to add cni-plugins as optional dependency for podman to make it more clear, that it can also be used.

[acxz]

Ah til about netavark.

Thanks for the response!

[xxxserxxx]

Optional dependencies are fine; I don't know the best way to document this, but it looks like several of us (from varying distributions) had to make our way to this ticket to figure out what the issue was. I'm not sure about others, but for me, something stopped working after an (minor version) upgrade.

Maybe it's coincidental that this impacted more than one distribution at a time, but if it wasn't then it might have been a breaking change which could have been communicated through a major revision bump.

I don't think netavark solves this, on Arch at least. netavark got installed with the change that broke my compose, and it was only fixed after installing cni-plugins.

[nolange]

Should'nt these config files be ignored when podman is using netavark? I dont use the cni plugins, but get that warning anyway as /etc/cni/net.d/87-podman-bridge.conflistis installed with podman 4.2.0 on debian.

[mheon]

Yeah, they should be ignored. Can you file a bug about this?

[acasanova99]

If you are using a debian based distro. This is the package you need to solve it.
You can just:

sudo apt install containernetworking-plugins
sudo apt update
podman system reset --force

And then everything should work.
Tested on Ubuntu v22.04 and podman v4.5.0

[jplatte]

@SirSkizo Thanks a lot, same thing worked for me on arch linux with the cni-plugins package! (I then also ran into #12637 but it somehow fixed itself shortly after)

[EvanCarroll]

I'm on Debian and also seeing this issue, though it seems clear to me I just need to install a dep, I wonder if a better message could be provided then,

WARN[0000] Error validating CNI config file /home/ecarroll/.config/cni/net.d/87-podman.conflist: [failed to find plugin "bridge" in path

And repeating that "failed to find plugin" for each plugin. Something along the lines of,

"CNI Plugins not detected, perhaps your distro packages them separately?"

would be far more useful. But this still raises the question why can I run all the commands to generate a kube.yaml by creating a pod, creating containers in the pod, and such and everything works fine. But when I podman generate kube myPod and then try to podman play the yaml that outputs, then I need these plugins. Why aren't they needed when I run the commands to create the pod through the CLI? That's a confusing interface, and it encourages distribution of podman commands rather than kube yaml (to shave the dep). I'm not sure if that's desired.

---

Actually I'm not sure what's going on here, seems I have all the plugin packages my dist offers,

sudo apt install golang-github-appc-cni-dev golang-github-containerd-go-cni-dev  golang-github-containerd-go-cni-dev
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
golang-github-appc-cni-dev is already the newest version (1.1.2-1).
golang-github-containerd-go-cni-dev is already the newest version (1.1.7-2).

And yet, I still get this error,

❯ podman kube play out.yaml
WARN[0000] Error validating CNI config file /home/ecarroll/.config/cni/net.d/87-podman.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]] 
WARN[0000] Error validating CNI config file /home/ecarroll/.config/cni/net.d/podman-default-kube-network.conflist: [failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "portmap" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "firewall" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]] 

WARN[0002] Failed to load cached network config: network podman-default-kube-network not found in CNI cache, falling back to loading network podman-default-kube-network from disk 
WARN[0002] 1 error occurred:
	* plugin type="tuning" failed (delete): failed to find plugin "tuning" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]
 
[starting container bf01101cf1462ba5f1f6c373e9e80ca29ccdb4a2dbc4d2392b4fc93c5ac5141a: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
[starting container bf01101cf1462ba5f1f6c373e9e80ca29ccdb4a2dbc4d2392b4fc93c5ac5141a: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] starting container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89: a dependency of container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89 failed to start: container state improper]
[starting container bf01101cf1462ba5f1f6c373e9e80ca29ccdb4a2dbc4d2392b4fc93c5ac5141a: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] starting container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89: a dependency of container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89 failed to start: container state improper starting container d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9: a dependency of container d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9 failed to start: container state improper]
[starting container bf01101cf1462ba5f1f6c373e9e80ca29ccdb4a2dbc4d2392b4fc93c5ac5141a: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] starting container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89: a dependency of container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89 failed to start: container state improper starting container d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9: a dependency of container d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9 failed to start: container state improper starting container fa55d7d24f3dae91d4a75d3022aae7f359c85f3c8500e9031ebaa65b08a2b0d0: a dependency of container fa55d7d24f3dae91d4a75d3022aae7f359c85f3c8500e9031ebaa65b08a2b0d0 failed to start: container state improper]
Pod:
fb0e030ac1fc38350d10d68cd41d827ecd152d08094082ee5613de81aaecd267
Containers:
8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89
fa55d7d24f3dae91d4a75d3022aae7f359c85f3c8500e9031ebaa65b08a2b0d0
d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9

starting container bf01101cf1462ba5f1f6c373e9e80ca29ccdb4a2dbc4d2392b4fc93c5ac5141a: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]
starting container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89: a dependency of container 8b0861393b3fbb6da66d95242098dbea856d0d145ec40ec8c87c6940a9efac89 failed to start: container state improper
starting container d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9: a dependency of container d6c959a8d30dd2991d9024ef92c53f0bf760da8e2ce4562c80470f6dd94c5eb9 failed to start: container state improper
starting container fa55d7d24f3dae91d4a75d3022aae7f359c85f3c8500e9031ebaa65b08a2b0d0: a dependency of container fa55d7d24f3dae91d4a75d3022aae7f359c85f3c8500e9031ebaa65b08a2b0d0 failed to start: container state improper

Error: failed to start 4 containers