systemd-sysext is causing issue to install the custom kernel modules

[kavinnath]

systemd-sysext is causing issue to install the custom kernel modules. Since i am using systemd-sysext image to install latest docker/containerd version and which mounts /opt as read only.

Ref:

systemd-sysext status
HIERARCHY EXTENSIONS SINCE
/opt none -
/usr docker-24.0.9

As per following doc if i try to install custom kernel modules on /opt then it will fail due to systemd-sysext.

So if i set modules path to some other writable location and use lowerdir=/usr/lib64/modules then it works fine. But the problem is /usr/lib64/modules are symlink and hence /etc/systemd/system/usr-lib64-modules.mount throws below exception.

Mount Ref:
sudo mount
-o "lowerdir=/usr/lib64/modules,upperdir=${modules},workdir=${modules}.wd"
-t overlay overlay /usr/lib64/modules

Exception logs:

usr-lib64-modules.mount: Mount path /usr/lib64/modules is not canonical (contains a symlink).
usr-lib64-modules.mount: Failed to run 'mount' task: Too many levels of symbolic links
usr-lib64-modules.mount: Failed with result 'resources'.
Failed to mount usr-lib64-modules.mount

Hence i tried to point source root i.e /usr/lib/modules and systemd service started but overlay mount is not working. Could you please help me out here ?

[pothos]

I don't fully understand what you mean with /opt becoming read-only. That would only be the case if you have sysext images that extend /opt.

Thanks for pointing out that the docs are outdated!
Instead of the custom mount action for /usr, with systemd-sysext you have to migrate it to be a sysext itself: Create a folder /var/lib/extensions/kernel-modules/usr/lib64/modules/ for your modules and a file /var/lib/extensions/kernel-modules/usr/lib/extension-release.d/extension-release.kernel-modules containing the line ID=_any. Afterwards they will be mounted on boot by systemd-sysext.service.
We should update https://www.flatcar.org/docs/latest/reference/developer-guides/kernel-modules/ Not sure if Before=local-fs.target would make sense as drop-in for systemd-sysext.service.

[kavinnath]

As suggested i created the sysext directory extension and got some exception.

systemd-sysext list
NAME TYPE PATH TIME
docker-24.0.9 raw /etc/extensions/docker-24.0.9.raw -
kernel-modules directory /var/lib/extensions/kernel-modules -

df -h /usr/lib64/modules/
Filesystem Size Used Avail Use% Mounted on
sysext 32G 12K 32G 1% /usr

And created development container and fails.

sudo systemd-nspawn
--bind=/usr/lib64/modules
--image=flatcar_developer_container.bin

Error:
flatcardevelopercontainer /usr/lib64/modules # touch a.txt
touch: cannot touch 'a.txt': Read-only file system

Note: Using Flatcar Container Linux by Kinvolk 3510.2.1.