conmon: container exec no longer returns exit code

[martinpitt]

Issue Description

Somewhere between Sep 2 and 4 the packages in podman-next broke healthchecking. This wasn't spotted by our per-PR cockpit-podman "revdeps" checks. A lot of PRs were merged red recently (understandably) because Testing Farm had major trouble in the past days, so many tests never actually started. Or something landed in podman-next that wasn't covered by podman revdeps, e.g. crun or similar?

• successful nightly run on Sep 2
• first failing nightly run on Sep 4

Steps to reproduce the issue

podman run --name sick -dt --health-cmd false --health-interval 5s docker.io/busybox

Describe the results you received

podman ps keeps claiming "healthy:

16ebccaf6eef  docker.io/library/busybox:latest  sh          4 minutes ago  Up 4 minutes (healthy)              sick

and journal keeps logging

container health_status 16ebccaf6eef8330d3c1206579834075950306da9f84c1121959078a1c706263 (image=docker.io/library/busybox:latest, name=sick, health_status=healthy, health_failing_streak=0, health_log=)

Describe the results you expected

Container should be unhealthy.

podman info output

host:
  arch: amd64
  buildahVersion: 1.42.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  - dmem
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.20250903100912264418.main.69.ge722bc2.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 95.35
    systemPercent: 2.34
    userPercent: 2.32
  cpus: 1
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: cloud
    version: "42"
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2047
  hostname: fedora-42-127-0-0-2-2201
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.16.3-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 510263296
  memTotal: 1126932480
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.17.0~dev-1.20250826181710402652.main.8.g9e014c5.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0-dev
    package: netavark-1.17.0~dev-1.20250825122416445226.main.22.g30c29d7.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.0-dev
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.20250903065722086996.main.35.g99b6cadd.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 524751d1ec4a83b21d3990d6d250da3f7fe4df95
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250805.g309eefd-2.fc42.x86_64
    version: |
      pasta 0^20250805.g309eefd-2.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 1126166528
  swapTotal: 1126166528
  uptime: 0h 15m 57.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 12802043904
  graphRootUsed: 2035154944
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.7.0-dev
  BuildOrigin: 'Copr: rhcontainerbot/podman-next'
  Built: 1756857600
  BuiltTime: Wed Sep  3 00:00:00 2025
  GitCommit: 6cc74679e2f85d3ad0be939f6f7368a59df6eb71
  GoVersion: go1.24.6
  Os: linux
  OsArch: linux/amd64
  Version: 5.7.0-dev

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Fedora 42 with podman-next COPR:

Upgrading:
 aardvark-dns                        x86_64 102:1.17.0~dev-1.20250826181710402652.main.8.g9e014c5.fc42     copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next   2.3 MiB
   replacing aardvark-dns            x86_64 2:1.16.0-1.fc42                                                updates                                                     2.3 MiB
 buildah                             x86_64 102:1.41.0-1.20250902202630851705.main.83.gf297289c7.fc42      copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next  30.9 MiB
   replacing buildah                 x86_64 2:1.41.3-1.fc42                                                updates                                                    30.9 MiB
 conmon                              x86_64 2:2.1.13-1.20250903100912264418.main.69.ge722bc2.fc42          copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next 170.7 KiB
   replacing conmon                  x86_64 2:2.1.13-1.fc42                                                fbf528756c684b6ab92bfe6b88c4f918                          166.3 KiB
 container-selinux                   noarch 102:2.241.0-1.20250901160001070837.main.3.g20230f8.fc42        copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next  72.6 KiB
   replacing container-selinux       noarch 4:2.241.0-1.fc42                                               updates                                                    72.6 KiB
 containers-common                   noarch 102:0.66.0~dev-1.20250903172927000225.main.14.ge06f072df1.fc42 copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next 132.1 KiB
   replacing containers-common       noarch 5:0.64.1-2.fc42                                                updates                                                   128.6 KiB
 containers-common-extra             noarch 102:0.66.0~dev-1.20250903172927000225.main.14.ge06f072df1.fc42 copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next   0.0   B
   replacing containers-common-extra noarch 5:0.64.1-2.fc42                                                updates                                                     0.0   B
 crun                                x86_64 102:1.23.1-1.20250903065722086996.main.35.g99b6cadd.fc42       copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next 577.2 KiB
   replacing crun                    x86_64 1.23.1-1.fc42                                                  updates                                                   573.1 KiB
 netavark                            x86_64 102:1.17.0~dev-1.20250825122416445226.main.22.g30c29d7.fc42    copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next  11.5 MiB
   replacing netavark                x86_64 2:1.16.1-1.fc42                                                updates                                                    11.4 MiB
 podman                              x86_64 102:5.7.0~dev-1.20250903205624286552.main.1516.6cc74679e2.fc42 copr:copr.fedorainfracloud.org:rhcontainerbot:podman-next  47.4 MiB
   replacing podman                  x86_64 5:5.6.0-1.fc42                                                 updates                                                    47.3 MiB

Additional information

No response

[Honny1]

I tried to reproduce this on commit 6cc74679e2f85d3ad (the latest on the main branch), but I was unsuccessful. (f41, arm64, build from source)

[Honny1]

I tested it with the podman package from Copper, and I was able to reproduce this. After installing the Copper RPMs, I am also able to reproduce it on a build from source.

[Honny1]

I suspect the problem isn't with the podman healthcheck, but with the container runtime itself. I've isolated the issue to the fact that the exit code from an executed process is not being propagated correctly. Here's a simple way to reproduce it:

podman exec sick /bin/sh -c 'exit 3'; echo $?
0 # expected 3

[Luap99]

I compiled conmon from source and then just run podman exec false and it didn't error so this is a conmon regression.
healtcheck is just an exec under the hood.

$ podman run --rm -d --name test quay.io/libpod/testimage:20241011 sleep 1000
87f9a2434319f4883ba9f5eafa1c0d1cf187e85f699e2351ec1d421fbb121776
$ podman --conmon /home/pholzing/CODE/conmon/bin/conmon exec test false
$ echo $?
0

Should be easy to bisect if needed.

cc @jnovy

[Luap99]

just a guess but containers/conmon#584 changed the exit handling there

[Honny1]

I have the same suspicion. I will try to find out where the problem is and fix it.

[martinpitt]

Thanks! That also explains why this wasn't caught by reverse test deps.