systemd-nspawn mode mounts /run on top of user's bind mounts

[chenxiaolong]

I'm trying to rebuild the kernel SRPM in mock and have it signed with my secure boot keys. Fedora's build servers currently do this by bind mounting /var/run/pesign into the mock chroot: http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bkernel/files/bkernel-site-defaults.cfg

I'm trying to replicate this in my local mock instance, but when using systemd-nspawn, /run is being bind mounted on top of the /run/pesign bind mount, making the pesign socket inaccessible in the chroot.

<mock-chroot> sh-4.4# mount | grep /run
tmpfs on /run/pesign type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /run/systemd/nspawn/incoming type tmpfs (ro,seclabel,mode=755)
<mock-chroot> sh-4.4# ls -l /var/run
lrwxrwxrwx. 1 root mock 6 Dec  5 08:15 /var/run -> ../run
<mock-chroot> sh-4.4# ls -l /run/pesign
ls: cannot access '/run/pesign': No such file or directory
<mock-chroot> sh-4.4# 

This does not happen with --old-chroot. I believe this could be fixed by having mock pass bind mounts to systemd-nspawn with --bind instead of mounting things itself.

Full logs: https://gist.github.com/chenxiaolong/c85d252d8459d7a09a0b82b3837ac7b2

[xsuchy]

Good idea. However, this is possible only for bindmounts. It will still not work for general mounts (eg. lvm or nfs). I will leave this open as the change to use --bind for bindmounts is great. But the general guidance is to use some general mountpoints (e.g. /mnt)

[chenxiaolong]

Thanks for the reply! It looks like this is affecting the ccache plugin too. It bind mounts the ccache directory to /tmp/ccache and then systemd-nspawn mounts its own tmpfs at /tmp.

[Conan-Kudo]

The proper thing to do is to move things from /run and /tmp to /mnt and reuse that for these persistent-ish things.

[xsuchy]

nod Although to refuse is probably too hard. But it would be nice to produce a strong warning if anyone tries to mount anything under /run or /tmp.

[belegdol]

Have there been any new developments? I needed to self-sign a kernel rpm and I had to use --old-root too.

[ozbenh]

Same here :-) Things like pesign (and associated RPM macros) more/less hard wire that the socket is in /var/run and I wouldn't be surprised if that is the only case.

I think it's probably worthwhile to fix the specific case of bind mounts as suggested initially (passing --bind) and leave the general case to "don't use /tmp or /run)

[nirik]

And... we just hit this in fedora, enabling systemd-nspawn. ;(

As a workaround, in /etc/mock/site-defaults.cfg, I think this will work:
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/var/run/pesign', '/var/run/pesign' ))
config_opts['nspawn_args'] = ['--capability=cap_ipc_lock','--bind=/var/run/pesign']

(well, only the second is needed for nspawn, but we also build stable branches with the same builders using old chroot).
I'll check and see if pesign might put it's socket in /var/lib/pesign or something.

[xsuchy]

Hmm, I really hoped that someone will move the pesign mounts to /mnt... but if the mountain will not come to Mohammed, .... we will try to do something with this in the Mock

[ozbenh]

I could try to submit patches to pesign to make the socket path an optional argument, but one would have to also change all the related RPM macros from hell, it's messy...

Note: For Amazon Linux I'm looking at using the pkcs11 proxy in p11-kit to "escape" the mock instead of using pesign in client/server mode, which is more versatile and can specify the socket path. Maybe Fedora wants to go down that path as well ?

[praiskup]

From Mock's perspective, there's not much difference between /run/pesign or /mnt/pesign. What am I missing? User either has to do one hack (tweak plugin_conf.bind_mount_opts) or two (tweak also nspawn_args). The only thing we could do about this is to provide a new option (say pesign = True|False) so this is a bit easier to configure for the users?

The proper thing to do is to move things from /run and /tmp to /mnt and reuse that for these persistent-ish things.

I'm not sure. Is the /run/pesign stuff on host supposed to be a persistent thing?

[frozencemetery]

I'm not sure. Is the /run/pesign stuff on host supposed to be a persistent thing?

It's a runtime socket created by the pesign daemon for communication with it (and not a filesystem).

[praiskup]

Yes, that's what I thought - AIUI we can hardly expect this to be present under /mnt?