Fails to create directory /mnt/rootmirror/sys

[PeterJCLaw]

Testing this on my Pi 3 under stretch I get the following lines in the log:

INFO:runusb: Found new mountpoint: /media/usb0
INFO:runusb: -> launched process
Spawning container media_usb0 on /mnt/rootmirror.
Press ^] three times within 1s to kill container.
Failed to create directory /mnt/rootmirror/sys: Read-only file system

After that the next messages are the usual scanning ones (Detected a change in the procfile etc.)

The lines my main.py should print don't appear, so I'm assuming this is catastrophic for the container.

[RealOrangeOne]

I'm not actually sure why it's trying to create /sys there, there's no reason it should. cc @prophile

[RealOrangeOne]

It's possible creating this is due to a change in how systemd-nspawn works. Perhaps we can use the version of systemd-nspawn found in jessie? Not exactly sure how we do this though myself?

[PeterJCLaw]

Progress so far:

• this did definitely work on at least some versions of Jessie, and appears to be a result of the upgrade to Stretch. I'm not sure what part of the upgrade is responsible though. We've moved from systemd-nspawn version 215 to 232, which could be related (though may be a red herring)
• removing the --read-only flag from running the container seems to resolve the issue; running ls -l / inside that container shows the right things at /, suggesting that (at least non-read-only) systemd-nspawn is happy with the bind-mounting of the root directory to /mnt/rootmirror
• nspawn: do not uid shift mount points if --read-only systemd/systemd#7122 suggests that there may be some chown shenanigans happening which we're seeing, though I'm not sure why the error message is about creating a directory

[PeterJCLaw]

Ok, after switching to using Google rather than DuckDuckGo for searching (term "systemd-nspawn --read-only") I found https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876103 which suggests that this is a general issue in stretch. In that report systemd/systemd#4693, specifically systemd/systemd@acbbf69, is suggested as the fix.

Unfortunately the conversation goes towards the changes being too invasive and suggesting using stretch-backports. I'm not sure we really want to be using systemd from backports (that does feel invasive), nor do I think we really want to be building our own systemd!

[RealOrangeOne]

I'm really surprised the issue is in systemd itself, rather than just systemd-nspawn. Using systemd from backports isn't ideal, no. But if it will work, it's probably our best shot! What are the chances of using systemd from stretch, and systemd-nspawn (systemd-container) from Jessie?

/mnt/rootmirror is mounted readonly, so assuming that stays readonly, the container cant mutate it anyway, thus making the --read-only flag mute?

[PeterJCLaw]

I don't think that using the version from Jessie is an option -- remember that systemd-nspawn is in a separate package in Stretch, so we'd need to transplant the whole of systemd.

[PeterJCLaw]

I think the next thing to try is backports. If that doesn't work, then maybe look at using our own package with just the patches we need applied on top of the maintained version (which is icky).

Failing that, we could look at moving back to Jessie entirely. [edit] The latest Jessie images appear to still be available from https://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2017-07-05/.