systemd-nspawn fails to --bind directories

[hmalzahn]

Submission type

• Bug report
• Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

232

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Arch Linux

In case of bug report: Expected behaviour you didn't see

systemd-nspawn fails to --bind=/tmp/.X11-unix into the container

In case of bug report: Unexpected behaviour you saw

none

In case of bug report: Steps to reproduce the problem

btrfs subvolume snapshot / /.nspawn ; systemd-nspawn -D /.nspawn --bind=/tmp/.X11-unix bash
In 231 this shows /tmp/.X11-unix in the container, in 232 /tmp is empty.
I've attached the 232 output with LOG_LEVEL=debug

nspawn.txt

[jwwalton]

Confirmed. Worked fine in 231 and before that.

[L3n41c]

I do have the problem as well.
It seems to impact only bind mounts inside /tmp. Bind mounts elsewhere in the filesystem are still working fine with v232.

It is as if the bind mounts were done prior to mounting the tmpfs on /tmp since unmounting /tmp reveals the missing bind mounts:

lenaic@ncelhuard:~/doc/devel/open-source/systemd ((v232) $%)$ find /tmp/foo/ -ls
    83555      0 drwxr-xr-x   2  lenaic   lenaic         60 déc.  3 07:48 /tmp/foo/
    86072      4 -rw-r--r--   1  lenaic   lenaic          7 déc.  3 07:48 /tmp/foo/hello.txt

lenaic@ncelhuard:~/doc/devel/open-source/systemd ((v232) $%)$ cat /tmp/foo/hello.txt 
Hello!

lenaic@ncelhuard:~/doc/devel/open-source/systemd ((v232) $%)$ sudo ./systemd-nspawn -bi ../mkosi/arch.raw --bind /tmp/foo
Spawning container arch.raw on /home/lenaic/doc/devel/open-source/systemd/../mkosi/arch.raw.

[…]

Arch Linux 4.8.11-1-ARCH (console)

arch login: root
Last login: Fri Dec  2 17:12:05 on console

[root@arch ~]# find /tmp/
/tmp/
/tmp/systemd-private-a70e15cb43e04b11864be0f43e6cc92f-systemd-resolved.service-kdXKg1
/tmp/systemd-private-a70e15cb43e04b11864be0f43e6cc92f-systemd-resolved.service-kdXKg1/tmp
/tmp/.Test-unix
/tmp/.font-unix
/tmp/.XIM-unix
/tmp/.ICE-unix
/tmp/.X11-unix

[root@arch ~]# umount /tmp 

[root@arch ~]# find /tmp/
/tmp/
/tmp/foo
/tmp/foo/hello.txt

[root@arch ~]# cat /tmp/foo/hello.txt 
Hello!

[root@arch ~]#

[L3n41c]

I have bisected systemd and it seems that this regression has been introduced by:
8492849: nspawn: let's mount(/tmp) inside the user namespace (#4340)

[evverx]

@L3n41c , indeed, 8492849 changed the mounts order:
Before: mount(tmp) -> mount(tmp/foo)
After: mount(tmp/foo) -> mount(tmp)

It seems to impact only bind mounts inside /tmp. Bind mounts elsewhere in the filesystem are still working fine with v232

Actually, that affects all "MOUNT_IN_USERNS"-paths:

systemd/src/nspawn/nspawn-mount.c

Line 545 in cb638b5

static const MountPoint mount_table[] = {

[evverx]

The quick fix is:

diff --git a/src/nspawn/nspawn-mount.c b/src/nspawn/nspawn-mount.c
index c9d5ac4..b6898e3 100644
--- a/src/nspawn/nspawn-mount.c
+++ b/src/nspawn/nspawn-mount.c
@@ -550,7 +550,7 @@ int mount_all(const char *dest,
                 { NULL,                  "/proc/sys",           NULL,    NULL,        MS_BIND|MS_RDONLY|MS_NOSUID|MS_NOEXEC|MS_NODEV|MS_REMOUNT, MOUNT_FATAL|MOUNT_IN_USERNS|MOUNT_APPLY_APIVFS_RO },                          /* ... then, make it r/o */
                 { "/proc/sysrq-trigger", "/proc/sysrq-trigger", NULL,    NULL,        MS_BIND,                                                               MOUNT_IN_USERNS|MOUNT_APPLY_APIVFS_RO },                          /* Bind mount first ...*/
                 { NULL,                  "/proc/sysrq-trigger", NULL,    NULL,        MS_BIND|MS_RDONLY|MS_NOSUID|MS_NOEXEC|MS_NODEV|MS_REMOUNT,             MOUNT_IN_USERNS|MOUNT_APPLY_APIVFS_RO },                          /* ... then, make it r/o */
-                { "tmpfs",               "/tmp",                "tmpfs", "mode=1777", MS_STRICTATIME,                                            MOUNT_FATAL|MOUNT_IN_USERNS },
+                { "tmpfs",               "/tmp",                "tmpfs", "mode=1777", MS_STRICTATIME,                                            MOUNT_FATAL },

                 /* outer child mounts */
                 { "tmpfs",               "/sys",                "tmpfs", "mode=755",  MS_NOSUID|MS_NOEXEC|MS_NODEV,                              MOUNT_FATAL|MOUNT_APPLY_APIVFS_NETNS },

[evverx]

Fix is waiting in #4824