unrestricted access to host's block devices after starting docker service in lxc container

[aston-r]

Dear Developers,

I have faced quite an unusual issue that after starting the docker service in lxc container there is access to host block devices despise the restrictions.

General Information:

• Linux centos 3.10.0-957.1.3.el7.x86_64 Prefix tests with lxc-test- #1 SMP Mon Nov 26 17:43:08 PST 2018 x86_64 x86_64 x86_64 GNU/Linux
• CentOS Linux release 7.6.1810 (Core)
• LXC 3.0.1(also checked on LXC 3.1.0)

In other words, I have the following setup:
Centos -> Centos LXC Container -> Docker Containers.

STR:
-create the container (lxc-create --template=download --name centos -- --dist centos --release 7 --arch amd64)
-attach to container and install docker-ce

• yum install -y yum-utils lvm2
• yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
• yum-config-manager --enable docker-ce-edge
• yum install -y docker-ce
• sed -i -e 's/=/sbin/modprobe overlay/=-/sbin/modprobe overlay/g' /usr/lib/systemd/system/containerd.service (due to ExecStartPre=/sbin/modprobe fails inside LXC  containerd/containerd#2772)

• create block device file (mknod -m 666 /dev/sdb1 b 8 17) in the container(use lsblk to the get the major and minor device type)

• check that restrictions from /usr/share/lxc/config/common.conf works as expected:

08:29:01 root@centos://
[#]> vgscan --mknodes
  Reading volume groups from cache.
  /dev/mapper/control: open failed: Operation not permitted
  Failure to communicate with kernel device-mapper driver.
  Check that device-mapper is available in the kernel.
  Incompatible libdevmapper 1.02.149-RHEL7 (2018-07-20) and kernel driver (unknown version).

08:29:06 root@centos://
[#]> mount /dev/sdb1 /mnt/
mount: permission denied

08:30:05 root@centos://
[#]> ll /dev/mapper/*
crw-------. 1 root root 10, 236 Dec 26 08:50 /dev/mapper/control

• start docker service(it fails, but it does not matter):

[#]> systemctl start docker.service
A dependency job for docker.service failed. See 'journalctl -xe' for details.

• check the restrictions again

[#]> mount /dev/sdb1 /mnt/
[#]> vgscan --mknodes
  Reading volume groups from cache.

[#]> ll /dev/mapper/
total 0
crw-------. 1 root root  10, 236 Dec 26 08:50 control
brw-rw----. 1 root disk 253,   0 Dec 26 08:51 lxbase-root
brw-rw----. 1 root disk 253,   1 Dec 26 08:51 lxbase-swap
brw-rw----. 1 root disk 253,   6 Dec 26 08:51 lxdata-template

[#]> df -h /mnt/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        10G  194M  8,9G   3% /mnt

[#]> echo TEST > /mnt/test

[#]> ll /mnt/test
-rw-r--r-- 1 root root 5 Dec 26 08:38 /mnt/test

As you can see after starting the docker service I have write access to block devices from the host hypervisor inside lxc container and can destroy the root partition

Could you please advice why restrictions from /usr/share/lxc/config/common.conf(in particular lxc.cgroup.devices.allow = b : m and lxc.cgroup.devices.allow = c : m) do not work after starting docker service?

I have attached lxc-config and output from cat /proc/1/mounts. general-info.zip

[Blub]

Probably because privileged containers are privileged enough to modify the limits after the fact. I'll have to see if my current version of #1302 has anything missing and if not recreate the PR for it with a rebased version.

[aston-r]

@Blub

Probably because privileged containers are privileged enough to modify the limits after the fact.

I also noticed that not only docker service can do this, but NM too:

16:09:04 root@centos://
[#]> yum install NetworkManager -y

16:10:02 root@centos://
[#]> vgscan --mknodes
  Reading volume groups from cache.
  /dev/mapper/control: open failed: Operation not permitted
  Failure to communicate with kernel device-mapper driver.
  Check that device-mapper is available in the kernel.
  Incompatible libdevmapper 1.02.149-RHEL7 (2018-07-20) and kernel driver (unknown version).
16:10:04 root@centos://

[#]> systemctl start NetworkManager
16:10:15 root@centos://

[#]> vgscan --mknodes
  Reading volume groups from cache.

16:11:56 root@centos://
[#]> cat /proc/$(ps -eo pid,comm | grep  NetworkManager |awk '{print $1}')/cgroup
11:memory:/lxc/centos
10:devices:/lxc/centos/system.slice/NetworkManager.service
9:hugetlb:/lxc/centos
8:cpuacct,cpu:/lxc/centos
7:blkio:/lxc/centos
6:perf_event:/lxc/centos
5:cpuset:/lxc/centos
4:pids:/lxc/centos
3:freezer:/lxc/centos
2:net_prio,net_cls:/lxc/centos
1:name=systemd:/lxc/centos/system.slice/NetworkManager.service

[brauner]

Privileged containers can technically escape their device limits in the legacy cgroup hierarchy. This won't be possible with the unified hierarchy which we support with 4.0 afaict.