Podman-compose or podman start -a [container] fails to netavark: No such file or directory (os error 2)

[saqeria]

Description

I'm on debian 11 and installed the latest podman, aardvark-dns, netavark listed in the testing repo. I tried setting up rootless podman according to these instructions. So far everything went smoothly. I switched to my designated user (non-root) for podman and ran podman-compose up with a docker-compose.yml file that I know is correct.

Steps to reproduce the issue:

1. Install podman [4.3.1+ds1-5+b1], podman-compose, Aardvark-dns [1.0.3-1+b1], Netavark [1.0.3-4] and slirp4netns [1.2.0-1].

2. Setup a rootless user according to these instructions and these.

3. Run podman-compose up or podman start -a [container name]

Describe the results you received:

podman@localhost: ~/$ podman start -a 1_ejabberd
Error: unable to start container 7a7fe0e7ec8ac98b32edd4ec8bbe4785062a486d1827334f9b42f54accc35dc3: netavark: No such file or directory (os error 2)
exit code: 125

Describe the results you expected:
Started container [ID] successfully

Additional information you deem important (e.g. issue happens only occasionally):
Full log output when running podman-compose:

$ podman-compose up
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
['podman', 'network', 'exists', 'xmpp_xmpp-network']
podman create --name=1_mariadb --security-opt no-new-privileges --label io.podman.compose.config-hash=123 --label io.podman.compose.project=xmpp --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=xmpp --label com.docker.compose.project.working_dir=/home/podman/compose_files/xmpp --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=mariadb -e TZ=RETRACTED -e MYSQL_ROOT_PASSWORD=RETRACTED -e MYSQL_DATABASE=RETRACTED -e MYSQL_USER=RETRACTED -e MYSQL_PASSWORD=RETRACTED -v /home/podman/volumes-docker/xmpp/mariadb/database:/var/lib/mysql -v /home/podman/volumes-docker/xmpp/mariadb/logs:/var/lib/mysql/mysql-bin --net xmpp_xmpp-network --network-alias mariadb -p 3306:3306 --restart unless-stopped ghcr.io/linuxserver/mariadb:latest
3c9a8a91f369b329d0cc2761cb13abf7ff1862af1e5b88919c6be0829280d672
exit code: 0
['podman', 'network', 'exists', 'xmpp_xmpp-network']
podman create --name=1_ejabberd --security-opt no-new-privileges --label io.podman.compose.config-hash=123 --label io.podman.compose.project=xmpp --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=xmpp --label com.docker.compose.project.working_dir=/home/podman/compose_files/xmpp --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=ejabberd -v /home/podman/volumes-docker/xmpp/ejabberd/config:/home/ejabberd/conf -v /home/podman/volumes-docker/xmpp/ejabberd/database:/home/ejabberd/database -v /home/podman/volumes-docker/xmpp/ejabberd/logs:/home/ejabberd/logs -v /home/podman/volumes-docker/xmpp/ejabberd/upload:/home/ejabberd/upload --net xmpp_xmpp-network --network-alias ejabberd -p 5222:5222 -p 5223:5223 -p 5269:5269 -p 5270:5270 -p 5280:5280 -p 5443:5443 -p 3478:3478 -p 5349:5349 --restart unless-stopped docker.io/ejabberd/ecs:latest
7a7fe0e7ec8ac98b32edd4ec8bbe4785062a486d1827334f9b42f54accc35dc3
exit code: 0
podman start -a 1_mariadb
Error: unable to start container 3c9a8a91f369b329d0cc2761cb13abf7ff1862af1e5b88919c6be0829280d672: netavark: No such file or directory (os error 2)
exit code: 125
podman start -a 1_ejabberd
Error: unable to start container 7a7fe0e7ec8ac98b32edd4ec8bbe4785062a486d1827334f9b42f54accc35dc3: netavark: No such file or directory (os error 2)
exit code: 125

Output of podman version:

$ podman version
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.4
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info:

$ podman info
host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.3+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.3, commit: unknown'
  cpuUtilization:
    idlePercent: 98.82
    systemPercent: 0.58
    userPercent: 0.6
  cpus: 4
  distribution:
    codename: bullseye
    distribution: debian
    version: "11"
  eventLogger: journald
  hostname: RETRACTED DOMAIN
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 65534
      size: 1
    - container_id: 1
      host_id: 110000
      size: 10001
    uidmap:
    - container_id: 0
      host_id: 109
      size: 1
    - container_id: 1
      host_id: 110000
      size: 10001
  kernel: 5.10.0-20-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 2443448320
  memTotal: 4644118528
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.5+dfsg-1+b1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.5
      commit: 54ebb8ca8bf7e6ddae2eb919f5b82d1d96863dea
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/109/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.1
  swapFree: 4831834112
  swapTotal: 4831834112
  uptime: 0h 36m 12.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 100813877248
  graphRootUsed: 4831887360
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/109/containers
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Package info:

podman/unstable,now 4.3.1+ds1-5+b1 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Debian 11 VPS

[ancebfer]

Install iptables package and try again.

containers/podman-compose#609

[Luap99]

looks like this is the same as #16958
This is a debian bug and not a podman upstream issue so I am closing this one