Cgroups mounts detection is not working inside LXC on CentOS 7 host

[p-himik]

The list of relevant errors from dockerd:

level=warning msg="Your kernel does not support cgroup memory limit"
level=warning msg="Unable to find cpu cgroup in mounts"
level=warning msg="Unable to find blkio cgroup in mounts"
level=warning msg="Unable to find cpuset cgroup in mounts"
level=warning msg="mountpoint for pids not found"
Error starting daemon: Devices cgroup isn't mounted

(container)$ cat /proc/self/mountinfo | grep cgroup
597 596 0:67 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs none rw,size=10240k,mode=755,uid=1000000,gid=1000000
598 597 0:52 /cgroup/blkio /sys/fs/cgroup/blkio rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
599 597 0:52 /cgroup/cpuacct,cpu /sys/fs/cgroup/cpuacct,cpu rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
600 597 0:52 /cgroup/cpuset /sys/fs/cgroup/cpuset rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
601 597 0:52 /cgroup/devices /sys/fs/cgroup/devices rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
602 597 0:52 /cgroup/freezer /sys/fs/cgroup/freezer rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
621 597 0:52 /cgroup/hugetlb /sys/fs/cgroup/hugetlb rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
622 597 0:52 /cgroup/memory /sys/fs/cgroup/memory rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
623 597 0:52 /cgroup/name=systemd /sys/fs/cgroup/systemd rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
745 597 0:52 /cgroup/net_prio,net_cls /sys/fs/cgroup/net_prio,net_cls rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
746 597 0:52 /cgroup/perf_event /sys/fs/cgroup/perf_event rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
788 597 0:52 /cgroup/pids /sys/fs/cgroup/pids rw,nosuid,nodev,relatime master:203 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other

(host)$ uname -a
Linux n4.mga.local 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

As you can see, the file system in this case is fuse.lxcfs whereas at https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/utils.go#L156 it's expected to find either cgroup2 or cgroup.
It's completely outside of my expertise but as far as I can tell, LXCFS is used in this case to provide "a container aware cgroupfs tree" because kernels < 4.6 do not have cgroup namespaces support.

[cyphar]

Yeah this looks like fun. It's strange we haven't seen this before -- given that Docker-in-LXC has been known to work for a long time. I'll Cc @brauner for his opinion on this one.

The main issue I see with just adding an other special-case for fuse.lxcfs is that we can't really be sure that fuse.lxcfs is actually what we think it is when setting up the container (other filesystem names are provided by the kernel).

[p-himik]

I've seen only a single mention of this specific case online. And the user decided just to switch the host OS from CentOS 7 to Ubuntu.
Here's an interesting part though - on Ubuntu it all works even on kernel 4.4. I have no idea why - probably it's some Ubuntu patches or something.

[ChethanSuresh]

Similarly,
I do not use systemd cgroup2 mounts. Therefore mount cgroup2 manually.

# mount -t cgroup2 cgroup2 /sys/fs/cgroup/
# cat /proc/self/mountinfo  | grep cgroup2 
24 22 0:22 / /sys/fs/cgroup rw,relatime - cgroup2 cgroup2 rw

According to

runc/libcontainer/cgroups/utils.go

Line 88 in 3e425f8

if (subsystem == "" && fields[9] == "cgroup2") || opt == subsystem {

fields[9] should be cgroup2, but due to my manual mount
fields[9] = rw Changing fields[8] works fine.
While fields[9] works in ubuntu due to additional field in mountinfo.

Not many would face the same issue, but /proc/self/mountinfo output may vary

[cloud-66]

The same issue
Host server Centos 7.9.2009 with 3.10.0-1160.31.1.el7.x86_64
LXD container Ubuntu 20.04
If i run command

cgroupfs-umount
cgroupfs-mount

Docker is working . cgroup remount in differenet options/path

before cgroupfs-umount

cgroupfs-mount

after cgroupfs-mount

But ,after reboot all groups remount in old way and docker doesn't start again
Maybe i should add commands before start lxd container. Or i have to generate another mount comand with more appropriate parameters, default options doesn't work

[cyphar]

@brauner Is lxcfs used to fake cgroupfs with certain kernels / distributions? We can treat fuse.lxcfs as though it were cgroupfs if that makes sense, but I was under the impression lxcfs was only used to mask /proc (EDIT: ah, is this used for cgroup namespaces?)?

[brauner]

@brauner Is lxcfs used to fake cgroupfs with certain kernels / distributions? We can treat fuse.lxcfs as though it were cgroupfs if that makes sense, but I was under the impression lxcfs was only used to mask /proc (EDIT: ah, is this used for cgroup namespaces?)?

A long long time ago on a kernel version far far away, LXCFS implemented userspace cgroup namespaces. So on kernels without cgroup namespaces LXCFS can be used to emulate them but nowadays that's rarely used since such kernels should be super old.