Unprivileged container: Failed to mount rootfs

[r10r]

Required information

• Distribution: archlinux
• git describe --long: lxc-4.0.0-1004-g46ad3c97a
• uname -a: Linux 5.9.11-arch2-1 Prefix tests with lxc-test- #1 SMP PREEMPT Sat, 28 Nov 2020 02:07:22 +0000 x86_64 GNU/Linux
• cat /proc/cmdline BOOT_IMAGE=../vmlinuz-linux root=/dev/xvda1 rw systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all initrd=../initramfs-linux.img
• cat /proc/self/cgroup : 0::/user.slice/user-0.slice/session-98.scope

Issue description

I want to start an unprivileged container but startup fails with the following errors.

• Full logs: lxc.log.txt

 cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:356 - Operation not permitted - Failed to load bpf program: 
 dir - storage/dir.c:dir_mount:153 - Permission denied - Failed to mount "/usr/local/var/lib/lxc/test-systemd3/rootfs" on "/usr/local/lib/lxc/rootfs"
 conf - conf.c:lxc_mount_rootfs:1284 - Failed to mount rootfs "/usr/local/var/lib/lxc/test-systemd3/rootfs" onto "/usr/local/lib/lxc/rootfs" with options "(null)"
 conf - conf.c:lxc_setup_rootfs_prepare_root:3260 - Failed to setup rootfs for
 conf - conf.c:lxc_setup:3395 - Failed to setup rootfs
 start - start.c:do_start:1261 - Failed to setup container "test-systemd3"
 sync - sync.c:sync_wait:36 - An error occurred in another process (expected sequence number 5)
 start - start.c:__lxc_start:2069 - Failed to spawn container "test-systemd3"

[root@k8s-cluster8-controller /]# cat /etc/subuid 
root:10000:100
[root@k8s-cluster8-controller /]# cat /etc/subgid 
root:10000:100

The container start just fine if I remove lxc.idmap settings.

# Template used to create this container: /usr/local/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release focal --arch amd64
# Template script checksum (SHA-1): cc9fb79513ac24572cd765110fc7811c708fe0f8
#
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed

lxc.arch = linux64
lxc.net.0.type = empty
lxc.rootfs.path = dir:/usr/local/var/lib/lxc/test-systemd3/rootfs
lxc.uts.name = test-systemd3

lxc.cgroup2.devices.deny = a
lxc.cgroup2.devices.allow = c *:* rwm
lxc.cgroup2.devices.allow = b *:* rwm

lxc.apparmor.profile = unconfined

lxc.log.file = /tmp/lxc.log
lxc.log.level = TRACE

lxc.idmap = u 0 10000 50
lxc.idmap = g 0 10000 50

[brauner]

Please show me:

ls -al /usr/local/var/lib/lxc/test-systemd3/
ls -al /usr/local/var/lib/lxc/test-systemd3/rootfs

[r10r]

Since I messed around with the permissions already I've created a fresh container.

[root@k8s-cluster8-controller ~]# ls -al /usr/local/var/lib/lxc/test-systemd4/
total 16
drwxrwx---  3 root root 4096 Feb 14 11:08 .
drwxr-xr-x  6 root root 4096 Feb 14 11:08 ..
-rw-r-----  1 root root 1086 Feb 14 11:08 config
drwxr-xr-x 17 root root 4096 Feb 12 08:50 rootfs

[root@k8s-cluster8-controller ~]# ls -al /usr/local/var/lib/lxc/test-systemd4/rootfs/
total 68
drwxr-xr-x 17 root root 4096 Feb 12 08:50 .
drwxrwx--x  3 root root 4096 Feb 14 11:13 ..
lrwxrwxrwx  1 root root    7 Feb 12 08:43 bin -> usr/bin
drwxr-xr-x  2 root root 4096 Apr 15  2020 boot
drwxr-xr-x  3 root root 4096 Feb 14 11:08 dev
drwxr-xr-x 61 root root 4096 Feb 14 11:08 etc
drwxr-xr-x  3 root root 4096 Feb 12 08:44 home
lrwxrwxrwx  1 root root    7 Feb 12 08:43 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Feb 12 08:43 lib32 -> usr/lib32
lrwxrwxrwx  1 root root    9 Feb 12 08:43 lib64 -> usr/lib64
lrwxrwxrwx  1 root root   10 Feb 12 08:43 libx32 -> usr/libx32
drwxr-xr-x  2 root root 4096 Feb 12 08:43 media
drwxr-xr-x  2 root root 4096 Feb 12 08:43 mnt
drwxr-xr-x  2 root root 4096 Feb 12 08:43 opt
drwxr-xr-x  2 root root 4096 Apr 15  2020 proc
drwx------  2 root root 4096 Feb 12 08:43 root
drwxr-xr-x  2 root root 4096 Feb 12 08:45 run
lrwxrwxrwx  1 root root    8 Feb 12 08:43 sbin -> usr/sbin
drwxr-xr-x  2 root root 4096 Feb 12 08:43 srv
drwxr-xr-x  2 root root 4096 Apr 15  2020 sys
drwxrwxrwt  7 root root 4096 Feb 14 11:16 tmp
drwxr-xr-x 13 root root 4096 Feb 12 08:43 usr
drwxr-xr-x 12 root root 4096 Feb 12 08:44 var

• Full log lxc.log.txt

[root@k8s-cluster8-controller ~]# lxc-create -n test-systemd4 --template download -- --dist ubuntu --release focal --arch amd64
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu focal amd64 (20210212_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

[root@k8s-cluster8-controller ~]# cat /usr/local/var/lib/lxc/test-systemd4/config 
# Template used to create this container: /usr/local/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release focal --arch amd64
# Template script checksum (SHA-1): cc9fb79513ac24572cd765110fc7811c708fe0f8
#
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed

lxc.arch = linux64
lxc.net.0.type = empty
lxc.rootfs.path = dir:/usr/local/var/lib/lxc/test-systemd4/rootfs
lxc.uts.name = test-systemd3

lxc.cgroup2.devices.deny = a
lxc.cgroup2.devices.allow = c *:* rwm
lxc.cgroup2.devices.allow = b *:* rwm

lxc.apparmor.profile = unconfined

lxc.log.file = /tmp/lxc.log
lxc.log.level = TRACE

lxc.idmap = u 0 10000 50
lxc.idmap = g 0 10000 50

[brauner]

Since I messed around with the permissions already I've created a fresh container.

[root@k8s-cluster8-controller ~]# ls -al /usr/local/var/lib/lxc/test-systemd4/
total 16
drwxrwx---  3 root root 4096 Feb 14 11:08 .
drwxr-xr-x  6 root root 4096 Feb 14 11:08 ..
-rw-r-----  1 root root 1086 Feb 14 11:08 config
drwxr-xr-x 17 root root 4096 Feb 12 08:50 rootfs

[root@k8s-cluster8-controller ~]# ls -al /usr/local/var/lib/lxc/test-systemd4/rootfs/
total 68
drwxr-xr-x 17 root root 4096 Feb 12 08:50 .
drwxrwx--x  3 root root 4096 Feb 14 11:13 ..
lrwxrwxrwx  1 root root    7 Feb 12 08:43 bin -> usr/bin
drwxr-xr-x  2 root root 4096 Apr 15  2020 boot
drwxr-xr-x  3 root root 4096 Feb 14 11:08 dev
drwxr-xr-x 61 root root 4096 Feb 14 11:08 etc
drwxr-xr-x  3 root root 4096 Feb 12 08:44 home
lrwxrwxrwx  1 root root    7 Feb 12 08:43 lib -> usr/lib
lrwxrwxrwx  1 root root    9 Feb 12 08:43 lib32 -> usr/lib32
lrwxrwxrwx  1 root root    9 Feb 12 08:43 lib64 -> usr/lib64
lrwxrwxrwx  1 root root   10 Feb 12 08:43 libx32 -> usr/libx32
drwxr-xr-x  2 root root 4096 Feb 12 08:43 media
drwxr-xr-x  2 root root 4096 Feb 12 08:43 mnt
drwxr-xr-x  2 root root 4096 Feb 12 08:43 opt
drwxr-xr-x  2 root root 4096 Apr 15  2020 proc
drwx------  2 root root 4096 Feb 12 08:43 root
drwxr-xr-x  2 root root 4096 Feb 12 08:45 run
lrwxrwxrwx  1 root root    8 Feb 12 08:43 sbin -> usr/sbin
drwxr-xr-x  2 root root 4096 Feb 12 08:43 srv
drwxr-xr-x  2 root root 4096 Apr 15  2020 sys
drwxrwxrwt  7 root root 4096 Feb 14 11:16 tmp
drwxr-xr-x 13 root root 4096 Feb 12 08:43 usr
drwxr-xr-x 12 root root 4096 Feb 12 08:44 var

• Full log lxc.log.txt

[root@k8s-cluster8-controller ~]# lxc-create -n test-systemd4 --template download -- --dist ubuntu --release focal --arch amd64
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu focal amd64 (20210212_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

[root@k8s-cluster8-controller ~]# cat /usr/local/var/lib/lxc/test-systemd4/config 
# Template used to create this container: /usr/local/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist ubuntu --release focal --arch amd64
# Template script checksum (SHA-1): cc9fb79513ac24572cd765110fc7811c708fe0f8
#
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed

lxc.arch = linux64
lxc.net.0.type = empty
lxc.rootfs.path = dir:/usr/local/var/lib/lxc/test-systemd4/rootfs
lxc.uts.name = test-systemd3

lxc.cgroup2.devices.deny = a
lxc.cgroup2.devices.allow = c *:* rwm
lxc.cgroup2.devices.allow = b *:* rwm

lxc.apparmor.profile = unconfined

lxc.log.file = /tmp/lxc.log
lxc.log.level = TRACE

lxc.idmap = u 0 10000 50
lxc.idmap = g 0 10000 50

Well, you have created a privileged container that's why your container doesn't start. All of the files in the container's rootfs are owned by host root, i.e. uid 0. According to your idmapping in the container's config they should be owned by uid and God instead. :)

[r10r]

Thanks! I've already played with the permissions (changed the permissions of the rootfs directory to 777)
but I didn't take the config and rootfs parent directory into account.

After adding execute permissions the the rootfs parent it works:

chmod o+x /usr/local/var/lib/lxc/test-systemd4

According to your idmapping in the container's config they should be owned by uid and God instead. :)

:D So is this the right way to do so ?

chown -R 10000:10000 /usr/local/var/lib/lxc/test-systemd4

[root@k8s-cluster8-controller ~]# chown -R 10000:10000 /usr/local/var/lib/lxc/test-systemd4
[root@k8s-cluster8-controller ~]# lxc-start -n test-systemd4
[root@k8s-cluster8-controller ~]# lxc-attach  -n test-systemd4
lxc-attach: test-systemd4: terminal.c: lxc_terminal_create_native: 924 Not a directory - Failed to open terminal multiplexer device
root@test-systemd4:~# ls -l /
total 44
lrwxrwxrwx   1 root   root       7 Feb 12 07:43 bin -> usr/bin
drwxr-xr-x   2 root   root    4096 Apr 15  2020 boot
drwxr-xr-x   5 root   root     400 Feb 14 10:52 dev
drwxr-xr-x  61 root   root    4096 Feb 14 10:52 etc
drwxr-xr-x   3 root   root    4096 Feb 12 07:44 home
lrwxrwxrwx   1 root   root       7 Feb 12 07:43 lib -> usr/lib
lrwxrwxrwx   1 root   root       9 Feb 12 07:43 lib32 -> usr/lib32
lrwxrwxrwx   1 root   root       9 Feb 12 07:43 lib64 -> usr/lib64
lrwxrwxrwx   1 root   root      10 Feb 12 07:43 libx32 -> usr/libx32
drwxr-xr-x   2 root   root    4096 Feb 12 07:43 media
drwxr-xr-x   2 root   root    4096 Feb 12 07:43 mnt
drwxr-xr-x   2 root   root    4096 Feb 12 07:43 opt
dr-xr-xr-x 257 nobody nogroup    0 Feb 14 10:52 proc
drwx------   2 root   root    4096 Feb 12 07:43 root
drwxr-xr-x  12 root   root     360 Feb 14 10:52 run
lrwxrwxrwx   1 root   root       8 Feb 12 07:43 sbin -> usr/sbin
drwxr-xr-x   2 root   root    4096 Feb 12 07:43 srv
dr-xr-xr-x  13 nobody nogroup    0 Feb 14 10:52 sys
drwxrwxrwt   7 root   root    4096 Feb 14 10:52 tmp
drwxr-xr-x  13 root   root    4096 Feb 12 07:43 usr
drwxr-xr-x  12 root   root    4096 Feb 12 07:44 var

[r10r]

What really tricked me is that it's not the permissions on rootfs thats missing it really is the executable bit on it's parent.

Tried chown -R 10000:10000 /usr/local/var/lib/lxc/test-systemd4/rootfs - but that doesn't work.

[brauner]

What really tricked me is that it's not the permissions on rootfs thats missing it really is the executable bit on it's parent.

Yeah, though we usually log that and it doesn't matter for privileged containers. But you also need to chown the rootfs to the correct permissions, otherwise your container won't be able to interact with the filesystem.

> perms /var/lib/lxc/
drwx--x--x 711 /var/lib/lxc/
root@wittgenstein|~
> perms /var/lib/lxc/f3/
drwxrwx--- 770 /var/lib/lxc/f3/
root@wittgenstein|~
> ls -al /var/lib/lxc/f3/
total 16
drwxrwx---  3 100000 100000 4096 Feb 13 16:08 .
drwx--x--x  8 root   root   4096 Feb 11 09:46 ..
-rw-r-----  1 root   root   1575 Feb 13 16:08 config
drwxr-xr-x 17 100000 100000 4096 Jan 30 08:48 rootfs
root@wittgenstein|~
> ls -al /var/lib/lxc/f3/rootfs/
total 68
drwxr-xr-x 17 100000 100000 4096 Jan 30 08:48 .
drwxrwx---  3 100000 100000 4096 Feb 13 16:08 ..
lrwxrwxrwx  1 100000 100000    7 Jan 30 08:43 bin -> usr/bin
drwxr-xr-x  2 100000 100000 4096 Apr 15  2020 boot
drwxr-xr-x  3 100000 100000 4096 Jan 31 21:30 dev
drwxr-xr-x 61 100000 100000 4096 Feb  8 20:37 etc
drwxr-xr-x  3 100000 100000 4096 Jan 30 08:45 home
lrwxrwxrwx  1 100000 100000    7 Jan 30 08:43 lib -> usr/lib
lrwxrwxrwx  1 100000 100000    9 Jan 30 08:43 lib32 -> usr/lib32
lrwxrwxrwx  1 100000 100000    9 Jan 30 08:43 lib64 -> usr/lib64
lrwxrwxrwx  1 100000 100000   10 Jan 30 08:43 libx32 -> usr/libx32
drwxr-xr-x  2 100000 100000 4096 Jan 30 08:43 media
drwxr-xr-x  2 100000 100000 4096 Jan 30 08:43 mnt
drwxr-xr-x  2 100000 100000 4096 Jan 30 08:43 opt
drwxr-xr-x  2 100000 100000 4096 Apr 15  2020 proc
drwx------  2 100000 100000 4096 Feb  8 20:37 root
drwxr-xr-x  2 100000 100000 4096 Jan 30 08:46 run
lrwxrwxrwx  1 100000 100000    8 Jan 30 08:43 sbin -> usr/sbin
drwxr-xr-x  2 100000 100000 4096 Jan 30 08:43 srv
drwxr-xr-x  2 100000 100000 4096 Apr 15  2020 sys
drwxrwxrwt  7 100000 100000 4096 Feb 13 22:39 tmp
drwxr-xr-x 13 100000 100000 4096 Jan 30 08:43 usr
drwxr-xr-x 12 100000 100000 4096 Jan 30 08:44 var

Is what I have here for example.

[r10r]

Thanks. I think I finally got it:

• The mapped user ID must have execute permissions for all directories along the path to and including the rootfs.
• File permissions for the rootfs have to be changed to the mapped userid.

Can you agree with that?

I would like to add this to the lxc.container.conf manpage and send a pull request. Ok ?

[brauner]

Yeah, sure.

[chackoj-1204]

Hi, has this documentation been added yet? If not, could I please be assigned this issue and complete it as a part of a project for the Virtualization course at UT Austin, where we contribute to an open-source virtualization repository? Thank you!

[stgraber]

I'm not seeing it in the current dev tree, so I think this still needs doing.

I'd probably recommend writing a paragraph on unprivileged LXC in the main lxc manpage (doc/lxc.sgml.in) which would then cover those two points.