Support a mutable mode for systemd-sysext/confext

[pothos]

Support a mutable mode for systemd-sysext and systemd-confext. For systemd-sysext this mutable mode will help traditional distro users on Debian or Fedora become accustomed to systemd-confext. See systemd/systemd#24864 for the current discussion.
For systemd-confext we could use this mutable mode for our /etc overlay mount, making /usr/share/flatcar/etc a syscfg directory and setting up a symlink to it from /usr/lib/confexts/.
The mount command we use is mount -t overlay overlay -o lowerdir=/sysroot/usr/share/flatcar/etc,upperdir=/sysroot/etc,workdir=/sysroot/.etc-work,redirect_dir=on,metacopy=off,noatime /sysroot/etc (TODO: check implications of redirect_dir enabled vs disabled).

We've noticed that systemd-tmpfiles causes upcopies for C, L, d, Z rules which may be something to fix if the sysext would ship a file that is also covered by a systemd-tmpfiles rules (which could even be indirect if /etc is set up for selinux relabeling with Z).

[t-lo]

Notes from our discussions with upstream. A note on expectation management: the below is strongly work-in-progress, nothing is committed to. This is a snapshot in time of our discussions with the systemd folks.

To ease migration of existing workloads to (ultimately) immutable /etc, /usr, and /opt as well as to provide an intermediate option that allows operating hybrid systems that support both traditional software deployments as well as sysext/confext, the system may support three modes:

1. Automatic mode - "merge" will auto-detect how to handle directories based on the presence of /[dir].local directories in the root filesystem.
   1. By default, there is no change in behavior. Overlay mounts are immutable.
   2. If, for any /etc, /usr, or /opt, if a directory /[dir].local exists (e.g. /etc.local for /etc ) then the overlay is mutable and uses the respective /[dir].local as its writable "upperdir". Changes in [dir] will be written to [dir.local].
   3. If /[dir].local is a symlink then the symlink's realpath is used as the writable upperdir. This includes symlinks to the actual mountpoint - if /[dir].local -> /[dir], then /[dir] itself is the writable upperdir and hence will retain user changes even after the confext/sysext is unmerged. One major use case of this scenario is shipping "default" / "vendor" configurations in a sysext which can be customised / overridden by the user.
2. Configuration file controlled: /etc/systemd/[sysext|confext].conf control the behavior of "merge". This overrides "Automatic mode".
   1. By default, there is no change in behavior. Overlay mounts are immutable.
   2. Via the configuration file (.ini format) the system's behavior can be set to
      1. immutable: default, no change. Set explicitly, it can be used to override [dir.local] auto-detection described above.
      2. mutable: mount target becomes "upperdir", is writable, and will retain user changes. This is the equivalent of the "symlink" use case 1.iii. Note that if the underlying mount point is read-only (e.g. /usr residing on a read-only partition) this option will not apply for that respective directory, i.e. that directory will remain read-only. To enable / retain writes, users may use option 2.ii.c (ephemeral) below, or option 1.ii ([dir].local) above.
      3. ephemeral: mount target is writable but uses a tmpfs as "upperdir", therefore changes are possible as long as the sysext is in use but will not be retained after unmerge.
3. Controlled by command line options. This overrides the configuration file.
   1. By default, there is no change in behavior. Overlay mounts are immutable.
   2. --immutable the default. Can be used explicitly to override mutable or ephemeral modes configured in the config file.
   3. --mutable similar to 2.ii.b above. The same special case handling for read-only mount target applies.
   4. --ephemeral similar to 2.ii.c above

[t-lo]

uapi-group pr here: uapi-group/specifications#78

Please reference this when discussing mutability with upstream systemd (PRs, issues, and otherwise).

[pothos]

(TODO: check implications of redirect_dir enabled vs disabled).

Some distros like Flatcar enable it by default through the kernel option even if the mount option isn't passed. I tried the "undefined" behavior and it's basically that when removing the original folder in the lowerdir, the upper folder won't show the contents of the lowerdir anymore but only those present in the moved folder of the upperdir. That's a sane behavior and I think we should always pass the redirect_dir=on mount option. Not passing it would result in upcopies when a folder is renamed even if it's directly renamed back, and the updates of the lowerdir won't be propagated to the upperdir.

[krnowak]

This PR brings in patches for mutable overlays into Flatcar: flatcar/scripts#1753. The patches are already upstreamed, should be a part of systemd v256 release.

[pothos]

The migration of /usr/share/flatcar/etc to be a confext should be good to go. Probably through a symlink in /usr/lib/confexts/flatcar-defaults/etc, and a symlink /var/lib/extensions.mutable/etc → /etc, and then calling systemd-confext --mutable=yes --root=/sysroot in bootengine instead of the custom mount command.

A follow-up note on the =import and =ephemeral-import modes when we use a symlink to the original directory for the mutable setting: Currently this is not supported (error is Not importing mutable directory for hierarchy /etc as a lower dir, because it points to the hierarchy itself) but it would be nice to allow users to have a read-only or ephemeral mount while letting the mutable i.e. original directory win against sysexts. The problem with =yes or =ephemeral is that when we convert /usr/share/flatcar/etc to be a confext, that would overlay the original /etc in these modes, losing user changes. When we make =import and =ephemeral-import work we have a sane setting that users could use for the goal of having a read-only /etc or a ephemeral /etc.