CentOS 7 containers on CentOS 7.6 LXD host reports process utilization incorrectly

[trenb]

I tried logging a thread here, but doesn't seem to be going anywhere - https://discuss.linuxcontainers.org/t/centos-7-x-containers-on-a-centos-7-6-lxc-host-providing-incorrect-cpu-utilization/4863/4

So far I've been able to reproduce this on multiple kernels, my own compiled lxc/lxd/lxcfs and the Fedora provided packages at https://copr-be.cloud.fedoraproject.org/results/ganto/lxc3/epel-7-x86_64/

The issue is that the containers report that they're using more CPU than they actually are. It's only a reporting issue, however it makes monitoring CPU utilization inside the container difficult, as it reports 40% or more utilization, on a completely idle container.

I can only reproduce this with CentOS 7 64 bit containers. I can also reproduce it with a container launched via lxc launch images:centos/7/amd64 testing

I've run out of things to test myself, and if anyone has any ideas or suggestions, that would be appreciated.

[trenb]

Also, if I stop lxcfs and ensure it's not running when I start the container, process utilization reporting is correct.

[trenb]

@tomponline Here's the github issue for the above thread on discuss.linuxcontainers.org.

[hallyn]

Also, if I stop lxcfs and ensure it's not running when I start the container, process utilization reporting is correct.

Do you mean you stop lxcfs, start the container, then start lxcfs?

[trenb]

Nope, I mean ensure that lxcfs does not run at all when I start the container. If lxcfs is running, process utilization isn't properly reported.

[hallyn]

How reliable is this for you? I just created a fresh bionic VM, installed lxc from ppa, created an unpriv container and ran top. I'm not seeing this.

ii  lxcfs                            3.0.3-0ubuntu1~18.04.1                      amd64        FUSE based filesystem for LXC
ii  lxc1                           3.1.0+master~2019060 all                  Transitional package - lxc1 -> lxc-utils

Do you know which specific lxcfs (/proc) file is wrong in your tests?

[trenb]

Press the space-bar repeatedly to force refreshes. You should see it pretty quickly. I can reproduce this easily, and @tomponline reproduced this on ubuntu 16.04 and centos 7 hosts.

[hallyn]

Which value are you seeing too high? I'm still not really seeing it (once in awhile top comes back at '100%' but that happens for me even when not in a container).

Do you see this whether or not your cpuset.cpus is limited? Top does scale used cpu time by number of real hw threads, so it's possible that reducing the number of cpus seen in /proc/cpuinfo is throwing things off, since we don't scale what's in /proc/$$/stat etc.

[hallyn]

(It's also possible that this is a bug in the load tracking calculations, but I don't see /proc/loadavg giving bad numbers)

[hallyn]

actually here /proc/loadavg always gives me 0's.

[trenb]

If you check https://discuss.linuxcontainers.org/t/centos-7-x-containers-on-a-centos-7-6-lxc-host-providing-incorrect-cpu-utilization/4863 there's a screenshot from graphite showing inflated CPU use on LXD containers. As well I have several outputs from top there.

When @tomponline is able to jump in here, hopefully he can provide more technical information.

[stgraber]

As per what's mentioned in 283, can you check for any such log messages on your system when the issue occurs?