'Docker containers' dashboard displays 'No data'

[jlegido]

First of all many thanks to all the people involved in this project for their time, I really appreciate it.

Steps to reproduce

0. Cleanup (paranoic)

WARNING: don't copy/paste this step commands unless you know what you are doing

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

docker volume prune -f; docker network prune -f

docker system prune -fa

1. Clone the git repository

git clone https://github.com/stefanprodan/dockprom 

2. Enter the directory

cd dockprom

3. Start docker containers

ADMIN_USER=admin ADMIN_PASSWORD=admin ADMIN_PASSWORD_HASH=JDJhJDE0JE91S1FrN0Z0VEsyWmhrQVpON1VzdHVLSDkyWHdsN0xNbEZYdnNIZm1pb2d1blg4Y09mL0ZP docker-compose up -d

Wait until you get back the shell prompt.

4. Open a browser and proceed to grafana URL

http://localhost:3000

5. Type username and password (admin, admin)

6. Type twice new password (12345, 12345)

7. Click 'Dashboards / Manage'

http://localhost:3000/dashboards

8. Click 'Docker Containers'

9. Click 'Last 15 Minutes' (top bar) and select 'Last 5 Minutes'

Expected behaviour

Graph 'Container CPU Usage' should display data

Found behaviour

Graph 'Container CPU Usage' displays 'No data'.

Errors

None

Context

In aforementioned dashboard below graphs are displaying data:

• Running Containers
• System Load
• I/O Usage
• Etc.

Docker version 20.10.8, build 3967b7d
docker-compose version 1.28.5, build c4eb3a1f

I would swear that during several tests I made it work, but now I'm not able to see data on the docker container graphs.

[nightah]

Are all the containers coming up clean/successfully?

What's the output of docker-compose logs cadvisor?

[jlegido]

@nightah

Are all the containers coming up clean/successfully?

Yes:

docker ps

5827ae75235f   caddy:2.3.0                        "caddy run --config …"   About a minute ago   Up About a minute             80/tcp, 443/tcp, 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp, 0.0.0.0:9090-9091->9090-9091/tcp, :::9090-9091->9090-9091/tcp, 2019/tcp, 0.0.0.0:9093->9093/tcp, :::9093->9093/tcp   caddy
adb74d36013e   gcr.io/cadvisor/cadvisor:v0.40.0   "/usr/bin/cadvisor -…"   About a minute ago   Up About a minute (healthy)   8080/tcp                                                                                                                                                                         cadvisor
e6582dc2147c   prom/node-exporter:v1.2.2          "/bin/node_exporter …"   About a minute ago   Up About a minute             9100/tcp                                                                                                                                                                         nodeexporter
2497f561e6da   prom/pushgateway:v1.4.1            "/bin/pushgateway"       About a minute ago   Up About a minute             9091/tcp                                                                                                                                                                         pushgateway
6b81672e0d75   prom/prometheus:v2.29.2            "/bin/prometheus --c…"   About a minute ago   Up About a minute             9090/tcp                                                                                                                                                                         prometheus
0e2d0608fc7e   grafana/grafana:8.1.2              "/run.sh"                About a minute ago   Up About a minute             3000/tcp                                                                                                                                                                         grafana
2ff503d1ad7b   prom/alertmanager:v0.23.0          "/bin/alertmanager -…"   About a minute ago   Up About a minute             9093/tcp                                                                                                                                                                         alertmanager

What's the output of docker-compose logs cadvisor?

Attaching to cadvisor
cadvisor        | W0906 06:17:19.932917       1 manager.go:160] Cannot detect current cgroup on cgroup v2
cadvisor        | W0906 06:17:20.003676       1 machine_libipmctl.go:62] There are no NVM devices!

Apparently cadvisor is working as expected:

http://172.25.0.3:8080/docker/

Displays several entries for the containers, and if I pick one of them:

http://172.25.0.3:8080/docker/e6582dc2147c3685cb1ceda0ea838389876045d4a6838b80bd102ece400d31e7

It displays graphs, such as CPU or RAM usage.

[nightah]

Can you check the logs for Prometheus too? If cAdvisor is exporting metrics then we need to make sure it's making it's way over to Prometheus so Grafana can actually display.

It would be worthwhile taking one of the queries that Grafana runs and running it directly on Prometheus too.

[jlegido]

@nightah I'm pretty sure the issue, if any, is in the dashboard, since I realize that another dashboard, 'Monitor Services', works as expected.

For instance graph 'Container CPU Usage' is displaying data, and all running containers appears there, which I assume is because grafana -> prometheus -> cadvisor chain is working as expected.

Is somebody else able to reproduce the issue?

[nightah]

Do you mean like this?

Do you actually have any other running containers other than the dockprom stack?

[jlegido]

@nightah as soon as I created a new docker container outside the docker compose stack the aforementioned grafana dashboard started showing data.

The weird thing is that couple of odd random docker container names appears as well.

Anyways closing this task, since "expected behaviour" was wrong.

Thanks for your time.

[nightah]

If you do not name a container Docker will give it a random name, this is also expected behaviour for Docker.

[jlegido]

If you do not name a container Docker will give it a random name, this is also expected behaviour for Docker.

The think is that performing:

docker ps -a

I was not able to detect any docker container which such a random name.

[nightah]

Interim build containers are also considered so it might have been a transient container.