cAdvisor incorrectly queries Podman using Pod IDs as Container IDs → INFO log spam

[ashish-7711-cse]

When running cAdvisor on a Podman-based system, cAdvisor periodically queries the Podman API to collect container metadata. During this process, cAdvisor appears to treat Pod IDs as Container IDs, resulting in Podman returning:

level=info msg="Request Failed(Not Found): no container with name or ID \"3f52b41ef23a0c29544eaccafa0e482f3c811430de35eb9c8d39d1ec8bb59623\" found: no such container"

These messages are logged every 60 seconds by the Podman systemd service and create significant log noise in /var/log/messages or journalctl.

We are creating pods using podman kube play command.

There is currently no way to filter out infra containers or Pod IDs from cAdvisor's runtime API calls.

# podman ps
CONTAINER ID  IMAGE                                           COMMAND               CREATED      STATUS                PORTS                                                   NAMES
d64e93249f5e  appliance/autosupport:7.0.0   /sbin/init            8 hours ago  Up 8 hours (healthy)                                                          autosupport
2ecc072b1795   appliance/etcd:3.6.4          /usr/local/bin/et...  7 hours ago  Up 7 hours                                                                    etcd
1cb4a03e87c9  appliance/svc/auth:7.0.0                            7 hours ago  Up 7 hours (healthy)                                                          authservice
053658f3ab48  appliance/registry:3.0.0      /etc/distribution...  7 hours ago  Up 7 hours                                                                    registry
05568e479d4c  /svc/mgmt:7.0.0                                 7 hours ago  Up 7 hours (healthy)                                                          mgmtserver
6cc225b5a627  svc/gateway:7.0.0        /usr/sbin/nginx -...  7 hours ago  Up 7 hours (healthy)  10.81.202.68:443->443/tcp, 10.81.202.68:1443->1443/tcp  webui
4c3a323d8800  lsvc/settings:7.0.0                             7 hours ago  Up 7 hours (healthy)                                                          settings
b29d328bf38a  localhost/podman-pause:4.9.4-rhel-1762969518                          7 hours ago  Up 7 hours                                                                    3f52b41ef23a-infra
e2b60393a755  localhost/podman-pause:4.9.4-rhel-1762969518                          7 hours ago  Up 7 hours                                                                    4f4aa37568d2-infra
3389b212793a  /svc/peer:7.0.0                                 7 hours ago  Up 7 hours (healthy)                                                          peer-pod-peer
77af637b2ba7   /appliance/netinsights:6.2.0                         7 hours ago  Up 7 hours (healthy)                                                          remotemgmt-nic
a1d5213e9ffc    flex/svc/gateway:7.0.0        /usr/sbin/nginx -...  7 hours ago  Up 7 hours (healthy)                                                          hostapi
5df8dcc5a80f  localhost/podman-pause:4.9.4-rhel-1762969518                          7 hours ago  Up 7 hours                                                                    65e843376519-infra
d04ff306a2e5   /appliance/rabbitmq:3.12.4-12                        7 hours ago  Up 7 hours (healthy)                                                          pubsub-rabbitmq
ee57e9f04417  /svc/pubsub:7.0.0                               7 hours ago  Up 7 hours (healthy)                                                          pubsub-server
8bd3c2f1b879  /appliance/prometheus:3.5.0    --config.file=/et...  7 hours ago  Up 7 hours                                                                    prometheus 

# podman pod list
POD ID        NAME        STATUS      CREATED      INFRA ID      # OF CONTAINERS
65e843376519  pubsub      Running     7 hours ago  5df8dcc5a80f  3
4f4aa37568d2  remotemgmt  Running     7 hours ago  e2b60393a755  2
3f52b41ef23a  peer-pod    Running     7 hours ago  b29d328bf38a  2

Pods and there corresponding infra containers

POD                                Infra Container
65e843376519           65e843376519-infra
4f4aa37568d2.           4f4aa37568d2-infra
3f52b41ef23a              3f52b41ef23a-infra

Environment

cAdvisor version v0.49.1

Podman version 4.9.4-rhel

OS: RHEL

Container runtime API: Podman Docker-compat API (/run/podman/podman.sock)

cAdvisor service ExecStart:

cadvisor \
  -docker=unix:// \
  -housekeeping_interval=30s \
  -raw_cgroup_prefix_whitelist="/machine.slice/libpod" \
  -store_container_labels=false \
  -disable_metrics=advtcp,app,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory_numa,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp \
  -enable_metrics=cpu,memory,network