nspawn: overlayfs cannot be the root file system right now

[Andrei-Pozolotin]

1. the great feature of --overlay= overlay fs support
   https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html

2. somehow does not allow to utilize generated overlay stack as container root fs mount
   (we tried many different things, nothing works for us)

3. additionally overlay fs support is missing from machine.nspawn
   https://www.freedesktop.org/software/systemd/man/systemd.nspawn.html

so the request is to 2-fold
a) support overlay as root fs
b) control overlay settings from *.nspawn

Hi all,
I came to this bug when trying to use an image as a base for several containers, to share base OS updates in particular.

The workaround i found is to use several overlays. My setup is a base fedora image mounted read-only and additional file hierarchy for running the Caddy webserver: /usr/bin/caddy and /etc/Caddyfile.

systemd-nspawn -D /var/lib/machines/fedora --read-only --overlay /var/lib/machines/fedora/etc:/var/lib/machines/caddy/base/etc:/var/lib/machines/caddy/overlay/etc:/etc --overlay-ro /var/lib/machines/fedora/usr/bin:/var/lib/machines/caddy/base/bin:/usr/bin --tmpfs /var -M caddy -b

It works pretty well even if I would really enjoy more help from systemd to share data across images.

I see Lennart renamed the bug removing the .nspawn part of it which is the one that interest me the most in this case. I have opened #4634 for this purpose.

Cheers,

[LionNatsu]

Docs says I can do it in this way:

systemd-nspawn -b -D "/home/lion/aroot" --overlay="/home/lion/aroot:/home/lion/overlay:/"

However, I always get this:

Spawning container aroot on /home/lion/aroot.
Press ^] three times within 1s to kill container.
Failed to move root directory: Invalid argument

On the other hand, If I use --overlay="...:...:/root" or --overlay="...:...:/usr" or any else, it runs well.

I do think this is a bug...

[kangsterizer]

Note that if you mount/umount the overlay manually as root filesystem and start the container there it works fine (of course - that's a work around until this works in systemd-nspawn)

[dteleguin]

Just my 2¢, I'm using systemd-230 and overlays to build multiple containers on top of base OS image. Base image and containers live in /srv/machines:

srv/
└── machines/
    ├── base/
    │   └── (OS image here)
    ├── machine1/
    │   └── (container-specific data)
    ├── machine2/
    │   └── (container-specific data)
    └── work/
        └── (workdir for overlayfs)

At the same time, var/lib/machines/machine* entries are just empty dirs, meant to be the targets for overlay mounts. First I've tried systemd-nspawn --overlay=...:

[root@host ~]# systemd-nspawn -D /var/lib/machines/machine1 --overlay=/srv/machines/base:/srv/machines/machine1:/var/lib/machines/machine1
Directory /var/lib/machines/machine1 doesn't look like it has an OS tree. Refusing.

The only remaining option was to mount overlays manually. Since my /srv lives on a separate partition, a specific mount order should be respected. This can be done either via /etc/fstab (systemd≥220):

overlay /var/lib/machines/machine1 overlay x-systemd.requires=/srv,lowerdir=/srv/machines/base,upperdir=/srv/machines/machine1,workdir=/srv/machines/work/machine1

or via /etc/systemd/system/var-lib-machines-machine1.mount unit:

[Unit]
After=srv.mount
Requires=srv.mount
Description=Mount overlay container for machine1

[Mount]
What=overlay
Where=/var/lib/machines/machine1
Type=overlay
Options=lowerdir=/srv/machines/base,upperdir=/srv/machines/machine1,workdir=/srv/machines/work/machine1

[Install]
WantedBy=multi-user.target

Of course this should be repeated for each remaining machine. I hope one day systemd-nspawn will support this layout out of the box, another step towards Docker Done Right™ ;)

[oxwivi]

I want to use my host operating system as the base image for containers (as described in referenced #9044), can I use the overlay method @ghost described in the second comment? I don't quite understand the syntax and purpose of those overlays, can anyone get me started?