New errors when using systemd-nspawnd on filesystem

[jam3st]

systemd version the issue has been seen with

246_rc2

For a long long time I have been able to mount a root filesystem using systemd-nspawnd which is mounted on a filesystem that isn't /root. With 246_rc I have been getting the error:

Failed to mount n/a (type n/a) on /run/host (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND ""): No such file or directory

[poettering]

This is not a useful bug report. I don't understand what the problem is supposed to be. Please fill in the bug submission form properly, it exists for a reason. i.e. we want to know how you specifically ran into the issue, and we need the command lines and stuff to reproduce the issue.

[jam3st]

Been using systemd-nspawn for two years and never saw anything being mounted in /run/host. It's useful when you want to test a few things out before sending them off to containers on the cloud (those that openvz and even kvm/xen).

I suspect that /run/host was never working because I don't see a /run/host in my host. There is nothing in the release that creates this. I don't think that this used to matter because prior to commit d64e32c systemd-nspawn didn't exit if that mount failed. Now it does. Is there some grand plan for /run/host? Why does it need to exist in the container?

Anyhow, this is how the container is run:

function runcont() {
	contpath="${staging_base}/${1}/"
	contname="${1}"
	case "${contname}" in
	"redacted") 
		iphost="redacted/24 broadcast redacted"
		ipgw="redacted"
		ip6host="redacted64"
		ip6gw="redacted"
		;;

	*)
		echo "Todo"
		exit 1
	esac
	if [ ! -d "${distFiles}" ] ; then
	    mkdir -p "${distFiles}"
	    chown portage:portage "${distFiles}"
        fi
	if [ ! -d "${buildDir}/${contname}" ] ; then
	    mkdir -p "${buildDir}/${contname}"
	    chown portage:portage "${buildDir}/${contname}"
        fi
	rm -f "${contpath}/etc/random-seed"
	/lib/systemd/systemd-random-seed save
	cp /run/random-seed ${contpath}/etc/
	shift 
	command='echo "nameserver redacted" > /run/resolv.conf ; '
	command+="ip addr add dev host0 ${iphost} ; ip link set host0 up ; ip route add default via ${ipgw} ; cd ; "
	command+="ip -6 addr add dev host0 ${ip6host} ; ip -6 route add default via ${ip6gw} ; cd ; "
	command+="mv /etc/random-seed /run/ ; "
	command+="dbus-uuidgen --ensure=/run/machine-id ;"
	command+="chmod 1777 /tmp ; "
	command+="exec ${*}"
	systemd-nspawn \
		--register=no \
		--tmpfs=/tmp \
		--tmpfs=/home \
		--tmpfs=/var/cache \
		--tmpfs=/var/run \
		--tmpfs=/var/log \
		--tmpfs=/var/tmp \
		--bind-ro=${kernel_git}:/tmp/linux-stable.git \
		--bind=${portage_dir}:/tmp/portage/ \
		--bind=${distFiles}:/tmp/distfiles \
		--bind=${buildDir}/${contname}:/tmp/build \
		--link-journal=no \
		--private-network \
		--network-veth \
		--setenv=HOME=/tmp \
		--setenv=SHELL=/bin/bash \
		--machine "build-${contname}" --directory "${contpath}" \
		/bin/bash --login -c "${command}"
}

if [ "$#" -ne 0 ]; then
    if [ "$#" -ne 1 ] ; then
        runcont ${*} 
    else 
        runcont ${*} /bin/bash
    fi
fi

[poettering]

/run is a tmpfs nspawn automatically mounts, which is writable. It always did that.

Since 246 it creates /run/host inside it as subdir, and it then creates a bind mount on itself on it. The bind mount is then re-mounted read-only, and that is apparently what fails. Which is highly irritating.

Do you have a MAC of some form running? selinux?

Any chance you can strace -s 500 -f -o log systemd-nspawn … the invocation this? and then provide the "log" output somewhere? it should tell us what precisely fails.

[poettering]

Hmm, or do you pre-mount /run on the host? or so? i don't get it?

/run must be writable to the container mgr...

[jam3st]

I remember getting this to work fairly early on because I had to look at the nspawn source, about 2 years ago going from memory the the projects that I was working on. Some things might not be right because of that but I doubt it.

This container is for checking some gentoo builds. I will run the strace and get back to you later on today, but I cannot see anywhere where I mount /run. The only thing that I do is start bash directly, not any kinds of init.

I will attach all the strace with the polling.

[jam3st]

Here you go. /run is writeable but nothing is mounted there. /mnt/work/os/ar and / are the same files (not mounted the same but staged before syncing using rsync)
log.txt

[poettering]

Hmm, so the strace shows that the /run/host mount point is first mkdir()'ed and then turned into a bind mount on itself. But then later, when we try to make it read-only it suddenly vanished, and we get ENOENT.

And I think I know why now. because of the --tmpfs=/var/run you add. We require that /var/run is a symlink to /run, thus you are overmounting /run there again, and thus make our stuff vanish.

Do things work if you kill that argument? (This was always broken... maybe we should actually refuse opration if you try something like that)

--tmpfs=/tmp you can remove too, nspawn does that anyway.

[jam3st]

Thanks. But, it would still be nice if it didn't fail because it cannot find an os-release or some other non-essential file for non booting systems.

[poettering]

It doesn't fail because it cannot find os-release. It fails because the preparations it did in /run suddenly disappeared... The handling of os-release worked fine, it's just that everything else it set up in /run disappeared...

Did it worked after you dropped the --tmpfs= switch?

[jam3st]

Yes, it mount /run/host after I removed that tmpfs switch. That was my bad. Sorry about that. But I just removed all those changes for /usr/lib/os-release because some of the stuff that I do doesn't have an os-release (too lazy to put it in the build scripts)

Thanks again,