setup-root: Create overlay mount for /etc to provide files from /usr

[pothos]

The existing tmpfile logic took care of folders that the ebuild keepdir directive wanted to exist on the OS. However, files and symlinks were not created, causing them to be missing if we didn't explicitly modify the ebuild files in coreos-overlay to use tmpfiles or patching of paths to be in /usr. We need a logic to provide /etc files from the current /usr partition without getting stale. This can be done best with an overlay mount which requires to keep the original /etc files under /usr.
Set up an overlay mount from the initrd after basic root fs preparation (ensure that the user db and machine-id file are as expected and create non-/etc tmpfiles). What we get with an overlay mount is essentially a similar behavior to a 3-way merge because as long as the user didn't change the files, the old version is replaced with the new version and as soon as the user did changes, that file is frozen and wins over the provided old (in case of a rollback) or new versions from /usr. It does not work on file lines but on whole file contents, yet that is also what rpm-ostree does to my knowledge.

How to use

Follow-up: I would like to add a boot cleanup action before setting up the mount, and it should remove duplicate files to keep the /etc folder clean and actually make use of the up-to-date lower files from /usr/share/flatcar/etc. ← But not a blocker from my point of view

Follow-up (mantle repo): Write a kola test that checks for unexpected upcopies (for now I did manual checks)

Follow-up: We should identify what downstream modifications in coreos-overlay we can drop now to have config files in /etc for legacy software.

Note: We should find out why systemd-tmpfiles causes upcopies for L and C entries and see if we can avoid it - maybe even upstream. For now I committed a change in the scripts repo to drop some rules after we used them to create /usr/share/flatcar/etc via a sed -i '/^[CLd] *\/etc\//d' /usr/lib/tmpfiles.d/* but it would be nice to fix this upstream.

Testing done

Jenkins kola tests passed, locally I checked the contents by mounting the image after a boot.

[pothos]

If the syscfg feature turns out to provide something similar, we can migrate - i.e. drop the custom mount command again and turn /usr/share/flatcar/etc into a syscfg folder by adding the metadata file and then setting up a symlink for it from the syscfg config folder.

[pothos]

I now see that the arm64 qemu test for coreos.update.badusr uncovers a continuous loop in the initrd instead of an emergency shell with a timeout. We might need to do some OnFailure thing for the initrd setup root unit or the initrd target in general.

Edit: Looks like a race condition, even - either in the check or the test or in the real behavior.

[JAORMX]

This is really nice! @pothos would this address the current issues with SELinux? that is that enabling custom modules requires removing the (hard?) links from /usr, which effectively limits how updates apply to SELinux policies.

cc @tormath1

[pothos]

Yes, we can review these downstream symlinks and config path changes and make use of /etc now where it makes sense. On the other hand we should also try to push upstream software to finally catch up with modern configuration requirements such as multiple search paths and drop-ins (A ready-made library exists https://github.com/openSUSE/libeconf and this is also captured in the update to the xdg basedir spec https://uapi-group.org/specifications/specs/base_directory_specification/).

[pothos]

Maybe it's better to add the boot cleanup action to duplicate files in a follow-up. This is ready to review now.

Here the list of contents that get upcopied/modified after boot as of now (/etc has many more files/dirs):

drwxr-x---.  2 root root system_u:object_r:auditd_etc_t:s0     4,0K 22. Feb 23:48 audit
drwxr-xr-x.  3 root root system_u:object_r:container_file_t:s0 4,0K 22. Feb 23:48 cni
-rwxr-xr-x.  1 root root system_u:object_r:etc_t:s0               0 22. Feb 13:21 environment
drwxr-xr-x.  2 root root system_u:object_r:etc_t:s0            4,0K 22. Feb 13:21 flatcar
-rw-r--r--.  1 root root system_u:object_r:unlabeled_t:s0        77 22. Feb 23:48 group
-rw-r-----.  1 root root system_u:object_r:unlabeled_t:s0        65 22. Feb 23:48 gshadow
-rw-------.  1 root root system_u:object_r:unlabeled_t:s0	144 22. Feb 23:48 .ignition-result.json
drwxr-xr-x.  2 root root system_u:object_r:etc_t:s0            4,0K 22. Feb 13:16 iscsi
-rw-r--r--.  1 root root system_u:object_r:etc_t:s0             25K 22. Feb 23:48 ld.so.cache
lrwxrwxrwx.  1 root root system_u:object_r:etc_t:s0              21 22. Feb 13:21 ld.so.conf -> ../usr/lib/ld.so.conf
-r--r--r--.  1 root root system_u:object_r:unlabeled_t:s0        33 22. Feb 23:48 machine-id
lrwxrwxrwx.  1 root root system_u:object_r:etc_t:s0              19 22. Feb 13:21 mtab -> ../proc/self/mounts
-rw-r--r--.  1 root root system_u:object_r:unlabeled_t:s0        82 22. Feb 23:48 passwd
-rw-------.  1 root root system_u:object_r:shadow_t:s0            0 22. Feb 13:21 .pwd.lock
lrwxrwxrwx.  1 root root system_u:object_r:etc_t:s0              34 22. Feb 13:21 resolv.conf -> ../run/systemd/resolve/resolv.conf
drwxr-xr-x.  2 root root system_u:object_r:selinux_config_t:s0 4,0K 22. Feb 13:21 selinux
-rw-r-----.  1 root root system_u:object_r:unlabeled_t:s0        40 22. Feb 23:48 shadow
drwxr-xr-x.  2 root root system_u:object_r:etc_t:s0            4,0K 22. Feb 23:48 ssh
drwxr-xr-x.  3 root root system_u:object_r:cert_t:s0           4,0K 22. Feb 13:21 ssl
drwxr-xr-x.  3 root root system_u:object_r:etc_t:s0            4,0K 22. Feb 13:15 systemd
drwxr-xr-x.  3 root root system_u:object_r:unlabeled_t:s0      4,0K 22. Feb 23:48 torcx
drwxr-xr-x.  2 root root system_u:object_r:etc_t:s0            4,0K 22. Feb 23:48 udev
-rw-r--r--.  1 root root system_u:object_r:unlabeled_t:s0	208 22. Feb 13:16 .updated

This is mostly expected (e.g., the user/group files get managed by flatcar-tmpfiles), odd cases are just things like udev where the 12 MB bin file gets copied over on each boot. I consider the goal of not getting slowly-outdated config files as reached.