/sys/devices/system/cpu/online not container aware

[ilhaan]

I have noticed that containers see all cores when I inspect /sys/devices/system/cpu/online, even though they have been restricted to a single core. /proc/cpuinfo seems to be reporting correctly.

For example, I have a container test:

root@server:~# lxc exec test -- grep -c "processor" /proc/cpuinfo 
1

Which is the expected output. However.

root@server:~# lxc exec test -- cat /sys/devices/system/cpu/online
0-55

I tested this using the following snap installed versions of LXD:

• 3.0.4 (from the 3.0/stable channel)
• 3.16 (from the 3.16/stable channel)

More info from my server:

root@server:~# lsb_release -a 
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic
root@server:~# uname -a 
Linux v45 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@server:~# snap --version 
snap    2.40
snapd   2.40
series  16
ubuntu  18.04
kernel  4.15.0-58-generic

Seems like there was a merge in May 2019 to address this. The comments on this merge make it seem like this should be enabled by default.

I discovered this while setting up a Kubernetes cluster with Rancher where I used LXD containers as nodes and posted an issue in the rancher repo. Rancher seems to be using /sys/ to enumerate CPU and memory resources on nodes, which is why I need LXCFS to fix my problem.

I also posted about this on the LXD forum.

Please let me know if you need me to provide additional information.

[stgraber]

Yeah, there is a bug in the generation of the online value...

root@shell01:~# grep lxcfs /proc/mounts
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
root@shell01:~# cat /sys/devices/system/cpu/online
0-7
root@shell01:~# grep ^Cpus_allowed_list /proc/self/status
Cpus_allowed_list:	5-6
root@shell01:~# grep ^processor /proc/cpuinfo 
processor	: 0
processor	: 1
root@shell01:~# stat -f /sys/devices/system/cpu/online
  File: "/sys/devices/system/cpu/online"
    ID: 0        Namelen: 255     Type: fuseblk
Block size: 512        Fundamental block size: 512
Blocks: Total: 0          Free: 0          Available: 0
Inodes: Total: 0          Free: 0
root@shell01:~# 

@brauner can you take a look? For some reason we appear to be hitting some fallback case where we get the host value rather than something which matches our task's cpuset configuration.

[ilhaan]

@stgraber Thanks for looking into this. I just noticed the following:

root@test:~# grep lxcfs /proc/mounts
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

I don't see /sys/devices/system/cpu/online listed above. Tested this with LXD 3.0.4 and 3.16 and got the same results.

[time-river]

I also find the problem, the following software use /sys/devices/system/cpu/online

1. # lscpu
2. nginx when use pid /run/nginx.pid;

Note: For nginx, it use sysconf(_SC_NPROCESSORS_ONLN); to get cpu number which the # strace is /sys/devices/system/cpu/online.

So, is there a plan to support virtual /sys/devices/system/cpu/online?

[stgraber]

There is support in lxcfs to virtualize /sys/devices/system/cpu/online as can be seen in my listing above on a suitably recent LXCFS, there is however an issue with the way it's rendered.

[yinhongbo]

@ilhaan @stgraber Please try to use the code of the latest master branch, I just submitted a PR may solve your problem.

[ilhaan]

@yinhongbo thanks for submitting the PR. Not sure how I can test this in a snap installed version of LXD. I'll try to figure it out, but would appreciate if you could give me some pointers on how to do this. Until then, I'll try poking around the LXD 3.17 snap.

[adamszen]

@ilhaan @stgraber Please try to use the code of the latest master branch, I just submitted a PR may solve your problem.

Still not working. I start lxcfs (latest master branch):

# lxcfs -l /var/lib/lxcfs/
mount namespace: 5
hierarchies:
  0: fd:   6: cpuset,cpu,cpuacct,blkio,memory,devices,freezer,perf_event,hugetlb,pids,rdma

All mounts are ok:

# mount | egrep "cgroup|lxcfs"
none on /cgroup type cgroup (rw)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,allow_other)

On host:

# cat /proc/cpuinfo  | grep processor
processor       : 0
processor       : 1
processor       : 2
processor       : 3
processor       : 4
processor       : 5
processor       : 6
processor       : 7
processor       : 8
processor       : 9
processor       : 10
processor       : 11

# cat /sys/devices/system/cpu/online
0-11

# cat /cgroup/cpuset.cpus
0-11

I do:

# echo "1-8" >/cgroup/lxc/cpuset.cpus
# cat /cgroup/lxc/cpuset.cpus
1-8

Then i start container (name: test):

# lxc-ls 
test

And do on host:

# echo "2" >/cgroup/lxc/test/cpuset.cpus
# cat /cgroup/lxc/test/cpuset.cpus
2

SSH into test and:

root@test:~# grep lxcfs /proc/mounts 
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

root@test:~# cat /proc/cpuinfo | grep processor
processor       : 0

OK, but:

root@test:~# cat /sys/devices/system/cpu/online
0-11

Not ok :(

It's something in cgfs_get_value function in bindings.c i thing - this funtcion is reading the wrong file?
Regards!

[yinhongbo]

@ilhaan @stgraber Please try to use the code of the latest master branch, I just submitted a PR may solve your problem.

Still not working. I start lxcfs (latest master branch):

# lxcfs -l /var/lib/lxcfs/
mount namespace: 5
hierarchies:
  0: fd:   6: cpuset,cpu,cpuacct,blkio,memory,devices,freezer,perf_event,hugetlb,pids,rdma

All mounts are ok:

# mount | egrep "cgroup|lxcfs"
none on /cgroup type cgroup (rw)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,allow_other)

On host:

# cat /proc/cpuinfo  | grep processor
processor       : 0
processor       : 1
processor       : 2
processor       : 3
processor       : 4
processor       : 5
processor       : 6
processor       : 7
processor       : 8
processor       : 9
processor       : 10
processor       : 11

# cat /sys/devices/system/cpu/online
0-11

# cat /cgroup/cpuset.cpus
0-11

I do:

# echo "1-8" >/cgroup/lxc/cpuset.cpus
# cat /cgroup/lxc/cpuset.cpus
1-8

Then i start container (name: test):

# lxc-ls 
test

And do on host:

# echo "2" >/cgroup/lxc/test/cpuset.cpus
# cat /cgroup/lxc/test/cpuset.cpus
2

SSH into test and:

root@test:~# grep lxcfs /proc/mounts 
lxcfs /proc/cpuinfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/diskstats fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/loadavg fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/meminfo fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/stat fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/swaps fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /proc/uptime fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
lxcfs /sys/devices/system/cpu/online fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

root@test:~# cat /proc/cpuinfo | grep processor
processor       : 0

OK, but:

root@test:~# cat /sys/devices/system/cpu/online
0-11

Not ok :(

It's something in cgfs_get_value function in bindings.c i thing - this funtcion is reading the wrong file?
Regards!

I think I know the reason for this problem. I am submitting a PR. Notify you after the merger.

[adamszen]

It works very well!

Sorry for the gif but it's easier to show me this ;)

screen-capture.gif

Thanks for this patch!

[stgraber]

Sounds like this got resolved with #307.