cadvisor won't give me information about specific container

[remo87]

I'm trying to run cadvisor using Docker version 27.4.1, build b9d17ea on Ubuntu 22.04. I'm trying to monitor several docker containers with prometheus but I don't see metrics segmented by docker container and when I look at the logs in the cadvisor contianer I see the following errors

manager.go:1116] Failed to create existing container: /docker/81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b: failed to identify the read-write layer ID for container "81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b". - open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b/mount-id: no such file or directory
2025-03-26 11:01:06 I0326 11:01:06.627503       1 manager.go:324] Recovery completed
2025-03-26 11:01:06 E0326 11:01:06.648703       1 manager.go:1116] Failed to create existing container: /docker/81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b: failed to identify the read-write layer ID for container "81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b". - open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b/mount-id: no such file or directory
2025-03-26 11:01:06 I0326 11:01:06.649375       1 cadvisor.go:179] Starting cAdvisor version: v0.52.1-0b675def on port 8080
2025-03-26 11:01:06 W0326 11:01:06.651612       1 manager.go:1169] Failed to process watch event {EventType:0 Name:/docker/81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b WatchSource:0}: failed to identify the read-write layer ID for container "81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b". - open /rootfs/var/lib/docker/image/overlayfs/layerdb/mounts/81fec498adee72467279a63af8457405a9fd38fed3f5bba838c026efdfdbb35b/mount-id: no such file or directory

Those errors referring to cadvisor container id but I get the same errors for all containers. I tried checking my local filesystem and don't see a folder called overlayfs but a folder overlay2.

The following is the part of the docker compose that defines cadvisor and prometheus

services:
...
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - 9090:9090
    command:
      - --config.file=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    depends_on:
      - cadvisor
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.52.1
    container_name: cadvisor
    privileged: true
    ports:
      - 8080:8080
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    environment:
      - GRUB_CMDLINE_LINUX="cgroup_enable=memory"
    network_mode: host
    restart: unless-stopped
    devices:
      - "/dev/kmsg"
    depends_on:
      - api
  ...

[remo87]

even if I try to query container_cpu_usage_seconds_total{name="cadvisor"} in prometheus there're no results

[lvaroqui-gpfw]

Got the exact same issue, did you find a solution?

[rjeczkow]

A workaround might be:

$ grep storage-driver /etc/docker/daemon.json
  "storage-driver": "overlay2"

Looks fine after restarting docker and cadvisor.