handle modified hardlinks to /sysroot

[cgwalters]

Today this:

FROM quay.io/fedora/fedora-coreos:stable
RUN touch /usr/bin/bash

When built using e.g. the redhat-actions/buildah-build@v2 Github action, seems to result in a tar stream that looks like this:

drwxr-xr-x root/root         0 1969-12-31 19:00 sysroot/
drwxr-xr-x root/root         0 1969-12-31 19:00 sysroot/ostree/
drwxr-xr-x root/root         0 1969-12-31 19:00 sysroot/ostree/repo/
drwxr-xr-x root/root         0 1969-12-31 19:00 sysroot/ostree/repo/objects/
drwxr-xr-x root/root         0 1969-12-31 19:00 sysroot/ostree/repo/objects/60/
-rwxr-xr-x root/root   1390000 2022-11-08 18:26 sysroot/ostree/repo/objects/60/feb13e826d2f9b62490ab24cea0f4a2d09615fb57027e55f713c18c59f4796.file
drwxr-xr-x root/root         0 2022-11-08 18:26 usr/
drwxr-xr-x root/root         0 2022-11-08 18:26 usr/bin/
hrwxr-xr-x root/root         0 2022-11-08 18:26 usr/bin/bash link to sysroot/ostree/repo/objects/60/feb13e826d2f9b62490ab24cea0f4a2d09615fb57027e55f713c18c59f4796.file

But that causes problems because when importing layered builds, we discard content in /sysroot entirely.

I think this has something to do with the use of the overlayfs driver versus vfs. I believe with the kernel overlayfs, modifying a hardlinked file will result in a copyup of just that single file; it explicitly will not mutate all targets of that link. From the docs:

2. "inode index"
   Enabled with the mount option or module option "index=on" or with the
   kernel config option CONFIG_OVERLAY_FS_INDEX=y.
   If this feature is disabled and a file with multiple hard links is copied
   up, then this will "break" the link. Changes will not be propagated to
   other names referring to the same inode.

I can think of two fixes:

• During ostree container commit, we can actually nuke everything in /sysroot - or at least just files whose timestamp is newer than the epoch. This would result in whiteout files in the serialized stream, which the import path will just ignore.
• Probably better: Actually handle this case at import time by rewriting the tar stream to have the link point to the first file name (hence decrementing the link count) or perhaps moving the file into e.g. /usr/lib/ostree/modified-objects/$digest and rewriting the links?

It may make sense to do both.