Some containers show zero CPU and RAM usage

[mind-overflow]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

When sending command docker stats, I see CPU/RAM/disk usage for all my containers.

Actual behavior

Some containers show CPU and RAM as 0% / 0 kB, however they are clearly using RAM as they are game servers. I can see they use about 1GB of RAM by looking at htop. Only some containers have this issue.

CONTAINER ID   NAME                                    CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
4d1922271d74   c5035006-9c11-439d-8f49-1ea20e7973cd    0.00%     0B / 0B               0.00%     0B / 0B           0B / 0B           0
f3ba840b924a   cc5592a5-e3f9-4bfe-a488-9b949636c556    0.00%     0B / 0B               0.00%     0B / 0B           0B / 0B           0
811fe368e520   remark42                                0.09%     27.25MiB / 7.775GiB   0.34%     21.6kB / 330kB    38.2MB / 34.3MB   11
64759ffdd93b   portainer                               0.21%     11.52MiB / 7.775GiB   0.14%     655kB / 9.75MB    586kB / 2.02MB    10
06cc86973a8d   commento_server_1                       0.05%     15.07MiB / 7.775GiB   0.19%     93.2kB / 252kB    14.5MB / 0B       10

Steps to reproduce the behavior

Unfortunately, I have no clue. One day it was reporting everything, then I came back after some weeks and it was like this.
However, what I can say is that if I look into Portainer (this container was not created with Portainer), while it does report usage on other working containers, it says cannot read property 'find' of null on the zero ones.

Output of docker version:

Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:52:25 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.7-0ubuntu1~20.04.1
  Built:            Wed Aug  4 19:07:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.2-0ubuntu1~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.2
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        

Output of docker info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 39
  Running: 35
  Paused: 0
  Stopped: 4
 Images: 50
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-81-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.775GiB
 Name: hosting.mind-overflow.net
 ID: MYIH:KHLT:Z7YH:DN77:I7L6:KKNT:733X:TBJO:BBM6:HOFZ:PVYU:VX2V
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)
I'm running on Ubuntu 20.04 on a KVM VPS (x64).

I noticed that a similar issue was reported on moby, however the suggested solution doesn't work (adding GRUB_CMDLINE_LINUX_DEFAULT="quiet cgroup_enable=memory swapaccount=1" to /etc/default/grub, updating grub and rebooting).

[kitkit201]

I too am running into this issue on some of our instances recently.. as of about last month when a new version of Docker was released to AWS.

This seems to start to happen about an hour or two when the tasks are up and running. Then the tasks slowly degrade over time and report 0 CPU%. This is causing a problem for us because we rely heavily on CPU to make scale up and down. When these tasks are falsely reporting metrics, it artificially increases the tasks that are needed to compensate for it.

$ docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 10
Running: 10
Paused: 0
Stopped: 0
Images: 5
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-1051-aws
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: aarch64
CPUs: 32
Total Memory: 61.69GiB
Name: ip-10-78-18-251
ID: WPF5:JA4K:IH3O:A7XZ:NNJC:W6YY:UXXL:SQQ4:JPZJ:Y3I5:KQIA:K6J4
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support

We have noticed the following logs that may offer clues. When we see this happen, that's when the instance stops reporting the CPU stats over to AWS ECS Agent.

2021-09-01T10:54:06.227155-07:00 ip-10-78-20-205 dockerd[4523]: time="2021-09-01T10:54:06.226889369-07:00" level=error msg="collecting stats for ff31d7a5898c00a5e4039fc49ebb8e2a6dd1dd0f956b55bdc18b3c5a88462d68: proto: wrong wireType = 0 for field CPU"

[kitkit201]

Any updates on this?

[cpuguy83]

I believe this is likely due to docker or containerd compiling with the wrong protos.

/cc @tianon maybe for the Ubuntu distro packages?

[tianon]

I don't think either package in Ubuntu explicitly generates the protos: 😕

• https://github.com/tianon/debian-docker/blob/debian/20.10.2-0ubuntu1_20.04.1/debian/rules
• https://github.com/tianon/debian-containerd/blob/debian/1.5.2-0ubuntu1_21.04.1/debian/rules

[FrankSealover]

Are you by chance running any game servers with the panel Pterodactyl?

[Winter]

Are you by chance running any game servers with the panel Pterodactyl?

@FrankSealover We are running Pterodactyl and are experiencing this particular issue as well. Did you ever manage to resolve the issue on your end?

[alexmorrisnz]

I'm getting this as well
Logs are full of these messages:

dockerd[2976]: time="2022-04-07T06:05:11.052432824Z" level=error msg="collecting stats for d52cf4cae54a91936b2407a078febd593b9363464d131945c40e02d769ad10d9: proto: wrong wireType = 0 for field Hugetlb"

Ubuntu 18.04.6
Docker version 20.10.14, build a224086
containerd containerd.io 1.5.11 3df54a852345ae127d1fa3092b95168e4a88e2f8

[cpuguy83]

Per investigation here, it seems to be related to setting --oom-kill-disable on the container: containerd/containerd#6700

I'm not sure if there's some other setting that triggers it.

[42wim]

Found the issue, moby is using outdated containerd/cgroups vendor.
I've got it fixed by updating https://github.com/moby/moby/blob/v20.10.15/vendor.conf

github.com/containerd/cgroups                       b9de8a2212026c07cec67baf3323f1fc0121e048 # v1.0.1
github.com/cilium/ebpf                              32458a752e0bcbc670691625f84e7ef6aef5655d # v0.4.0