Live system multiple overlays

[mid-kid]

Describe the enhancement
I want to be able to mount multiple root=live: squashfs filesystems on top of eachother.
This would allow me to add custom modules to my liveUSB without rebuilding the whole thing, and having one image that I can reuse across multiple variants.
Though primarily the purpose is making the customizations more easily introspectable.

There's been a previous issue and PR for this, but it fell out of favor: #686 #687

[LaszloGombos]

@mid-kid How do you feel about using systemd-sysext or you prefer a solution not using systemd ?

[mid-kid]

First time hearing of it, but it doesn't seem to be what I want. Not only because the image I'm building doesn't have systemd, but also simply because I intend to add files to /var and /etc, and both of these directories should remain writable through a tmpfs at runtime, like dracut sets them up to be.

That said, you're right in that it doesn't need to be handled in the initrd stage, I could do it in an early stage of booting after the rootfs kicks in. The thing is that I'd have to undo the dracut mounts which might break whatever dracut's doing to mount a persistent overlay.

I wonder, is anyone using the systemd-sysext program in a livecd context? I wonder what that'd look like, honestly.

[LaszloGombos]

Somewhat related, dracut can generate initramfs that prepares an overlay and copies over files from /updates - see https://github.com/dracutdevs/dracut/blob/master/modules.d/90dmsquash-live/apply-live-updates.sh .

CC @FGrose

[mid-kid]

Oh! That looks like a possible solution. It'd require more ram, but that's acceptable for small customizations. I'm just unable to find any documentation about it.

[LaszloGombos]

Oh! That looks like a possible solution. It'd require more ram, but that's acceptable for small customizations. I'm just unable to find any documentation about it.

There is a bit of information in the original commit message (which is over 10 years old) - f116ca1

The /updates directory needs to be in the root of the initramfs.

This can be used in combination of the idea is that some boot-loaders (e.g. grub) allows to specify more than one initramfs files - see https://www.gnu.org/software/grub/manual/grub/grub.html#index-initrd

So instead of several squashfs files, you would have several cpio files - maybe one or more with optional /updates/*

If this is working for you, than we would welcome contributions to improve the documentation (or perhaps even dracut tests) :-)

[mahdiaqallal]

Thanks @LaszloGombos

The /updates directory needs to be in the root of the initramfs.

And for the documentation you are referring to:

add apply-live-updates.sh to pre-pivot hook

If we're about to start a Live image (i.e. if /dev/mapper/live-rw
exists) this script will take any files found in /updates (inside the
initramfs!) and and copy them into $NEWROOT.

This allows for hotfixes to be applied to existing Live images without
rebuilding the entire image.

Could you please elaborate on how one should be taking advantage of apply-live-updates.sh ? so that "dracut generates an intramfs that prepares an overlay and copies over files from /updates" ?

Let's say I've downloaded Fedora Workstation 39 Live ISO onto my LiveUSB and wanting to use this method to include openssl or add custom modules like @mid-kid 's or insert custom files to be included in the Live System:

• how to create a /updates directory in the root of the initramfs to be updated ?
• what should be the format of files inside /updates : .img files ? plain files ?
• what would be the next steps ?

[mid-kid]

The idea is to make a cpio image with just an /updates directory, and chain it when loading the linux kernel (linux supports stacking multiple initrd files on top of eachother). The files then get copied from /updates into the tmpfs overlay of the running system, effectively overriding whatever is in the squashfs/base image.
The only disadvantage of this IMO is that while the files are compressed while in the cpio image (as initrd files may be compressed), as soon as it copies them to the tmpfs they're uncompressed and stuck in RAM instead of loaded on-demand. This increases the RAM usage by the size of the files updated, even if they don't end up being used.

Anyway, I might use this soon as I found a computer on which my live system won't run unless I bump the kernel... might add an early-boot script to bind-mount a new /lib/modules instead of sticking it all into tmpfs though.