No usage metrics for Docker resource usage

[pablomendezroyo]

A few users have reported already a problem with the docker grafana dashboard, saying that it is not displaying data. More info in this forum post,

When trying to visualize the catvisor at URL: http://dappnode-exporter.dappnode:8080/containers/, it throws the following message:

failed to get container "/" with error: unable to find data in memory cache

Error logs:

W0126 16:32:25.252736       8 container.go:526] Failed to update stats for container "/system.slice/systemd-udevd.service": failed to parse memory.usage_in_bytes - open /sys/fs/cgroup/system.slice/systemd-udevd.service/memory.usage_in_bytes: no such file or directory, continuing to push stats
W0126 16:32:26.760486       8 container.go:526] Failed to update stats for container "/init.scope": failed to parse memory.usage_in_bytes - open /sys/fs/cgroup/init.scope/memory.usage_in_bytes: no such file or directory, continuing to push stats
W0126 16:32:27.575993       8 container.go:526] Failed to update stats for container "/system.slice/containerd.service": failed to parse memory.usage_in_bytes - open /sys/fs/cgroup/system.slice/containerd.service/memory.usage_in_bytes: no such file or directory, continuing to push stats
W0126 16:32:27.762294       8 container.go:526] Failed to update stats for container "/system.slice/docker-d554732b6d34836c7af1cb17bc9b2f1410d371dfef390839235f616551697ad0.scope": failed to parse memory.usage_in_bytes - open /sys/fs/cgroup/system.slice/docker-d554732b6d34836c7af1cb17bc9b2f1410d371dfef390839235f616551697ad0.scope/memory.usage_in_bytes: no such file or directory, continuing to push stats
W0126 16:32:27.769149       8 container.go:526] Failed to update stats for container "/user.slice/user-1000.slice": failed to parse memory.usage_in_bytes - open /sys/fs/cgroup/user.slice/user-1000.slice/memory.usage_in_bytes: no such file or directory, continuing to push stats

After investigating this problem there might be two possible problem sources, both of them related to docker:

1. Docker version: it works on less than 19. Not working on 20.10.2

2. Users with dappnode instlled from an older version (core 0.2.39 or so): docker was installed in a completely different way, so the following path /usr/src/dappnode/bin/docker exists and this may be breaking how the catvisor read the logs from docker containers.

[pablomendezroyo]

There have been reported a few issues between docker and debian versions, more info about this incompatibility can be found in this [github comment]. (dappnode/DNP_DAPPMANAGER#462 (comment)).

Possible reasons ( updated ):

1. Docker might be installed in the wrong way by some users, important stuff such as cgroups is not "well" installed.
2. docker version not compatible with debian version.

Possible solutions:

1. Find the correct docker version for the debian version a user is running on the dappnode.
2. A users points that he needed to install cgroups manually to make docker working after an update. forum post

[pablomendezroyo]

UPDATE:

• Possible fix: try updating upstream version
• Possible problem: cadvisor might deppend on the docker version, dappnode exporter package does not know docker host machine version

The upstream registry might be not mantainned anymore, last push was on 29/02/2020: https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/cadvisor?gcrImageListsize=30

The new upstream registry might be (last push on 09/03/2021) https://console.cloud.google.com/gcr/images/cadvisor/GLOBAL/cadvisor?gcrImageListsize=30