feat(install): defer LUKS encryption to first boot

[andrewdunndev]

Summary

Defer LUKS encryption to first boot instead of running cryptsetup inside the install container. This eliminates the IPC namespace semaphore deadlock (#2089) and the shim/PCR mismatch problem (#421) in one change, since TPM2 binding happens on real hardware with real firmware state.

Prior art: openSUSE disk-encryption-tool, which ships a production implementation of first-boot encryption using the same cryptsetup reencrypt --encrypt mechanism.

Problem

bootc install to-disk --block-setup tpm2-luks runs cryptsetup luksFormat, systemd-cryptenroll, and cryptsetup luksOpen inside the install container. This has two problems:

1. IPC namespace deadlock (install to-disk --block-setup tpm2-luks hangs: libdevmapper udev cookie semaphore deadlock in container IPC namespace #2089): libdevmapper uses SysV semaphores to coordinate with udevd. Inside a container with an isolated IPC namespace, udevd on the host cannot see the container's semaphores, causing luksOpen and luksClose to hang on semop().

2. Shim/PCR mismatch (install to-disk with LUKS + TPM broken #421): TPM2 enrollment during install binds to the container's firmware state, not the installed system's firmware. On first real boot, the PCR values differ and auto-unlock fails.

Approach

Install time: Write an unencrypted root partition with the filesystem created 32MB smaller than the partition. Add rd.bootc.luks.encrypt=tpm2 to the kernel command line. No cryptsetup calls, no devmapper, no TPM2.

First boot: A dracut module (51bootc) installs a systemd service that runs before sysroot.mount. The service calls cryptsetup reencrypt --encrypt --reduce-device-size 32M to encrypt the root partition in-place using the reserved 32MB for the LUKS2 header. It then enrolls TPM2 via systemd-cryptenroll, writes /etc/crypttab, and updates the BLS entry with rd.luks.uuid / rd.luks.name kargs.

The root=UUID=<ext4-uuid> karg does not change. Once systemd-cryptsetup unlocks LUKS on subsequent boots, the ext4 UUID inside becomes visible and root= resolves normally.

Changes

crates/lib/src/install/baseline.rs

• Replace the Tpm2Luks arm: remove all cryptsetup/cryptenroll calls
• Add mkfs_with_reserve() that creates the filesystem smaller than the partition (ext4: block count arg, XFS: -d size=, btrfs: -b)
• Set rd.bootc.luks.encrypt=tpm2 karg instead of luks.uuid
• Set luks_device = None (no luksClose needed)

crates/initramfs/dracut/module-setup.sh

• Install bootc-luks-firstboot.sh and .service into the initramfs
• Install cryptsetup, systemd-cryptenroll, and dependencies
• Add dm_crypt kernel module
• Create sysroot.mount.requires symlink for service ordering

crates/initramfs/luks-firstboot/bootc-luks-firstboot.sh (new)

• First-boot encryption script (190 lines)
• Parses rd.bootc.luks.encrypt karg from /proc/cmdline
• Idempotent: checks cryptsetup isLuks before encrypting
• Encrypts via cryptsetup reencrypt --encrypt --reduce-device-size 32M
• Enrolls TPM2 via systemd-cryptenroll --tpm2-device=auto
• Generates recovery key via systemd-cryptenroll --recovery-key
• Updates /etc/crypttab and BLS entries

crates/initramfs/bootc-luks-firstboot.service (new)

• Runs Before=sysroot.mount in the initrd
• ConditionKernelCommandLine=rd.bootc.luks.encrypt
• OnFailure=emergency.target (drops to shell on failure)

Makefile

• Install the first-boot script to /usr/lib/bootc/

Testing

Full end-to-end validation on GCP n2-standard-8 with nested KVM, Fedora 42 (cryptsetup 2.8.4, systemd 257.11, QEMU 9.2.4 + OVMF + swtpm).

Build verification

Test	Result
cargo check	PASS
cargo build --release	PASS (4m05s)
cargo clippy -p bootc-lib	PASS (no new warnings)

Install verification

Installed with patched bootc binary via bootc install to-disk --block-setup tpm2-luks --filesystem ext4 to a 20GB disk.

Test	Result
Root partition not LUKS after install	PASS
Filesystem 32MB smaller than partition	PASS (33,554,432 bytes reserved)
rd.bootc.luks.encrypt=tpm2 in BLS entry	PASS
Separate /boot partition created	PASS
No cryptsetup calls during install	PASS (verified via serial console)

Encryption verification

Manually encrypted the installed root partition using the same cryptsetup reencrypt --encrypt --reduce-device-size 32M command the first-boot script uses.

Test	Result
20GB root encrypted in-place	PASS (110 seconds)
e2fsck clean after encryption	PASS
ostree deployment data intact	PASS
crypttab written to ostree deploy	PASS
BLS entry updated with rd.luks.uuid	PASS

Boot verification

Booted the encrypted system in QEMU with swtpm (vTPM 2.0). The initramfs was patched to include the first-boot dracut module with systemd-cryptsetup support.

Test	Result
systemd-cryptsetup@cr_root.service started	PASS
Passphrase prompt displayed	PASS
LUKS unlocked via passphrase	PASS
/dev/mapper/cr_root device created	PASS
sysroot.mount succeeded (ostree root)	PASS
System booted to login prompt	PASS

Serial console output (key lines):

Starting systemd-cryptsetup@cr_root.service - Cryptography Setup for cr_root...
Please enter passphrase for disk root (cr_root):
Finished systemd-cryptsetup@cr_root.service - Cryptography Setup for cr_root.
Reached target blockdev@dev-mapper-cr_root.target - Block Device Preparation for /dev/mapper/cr_root.
Mounted sysroot.mount - /sysroot.
...
fedora login:

Encryption mechanism validation (independent)

Tested cryptsetup reencrypt --encrypt --reduce-device-size 32M independently across multiple filesystem types and sizes.

Filesystem	Size	Time	Data Integrity	Result
ext4	2GB	10s	MD5 verified	PASS
XFS	2GB	12s	MD5 verified	PASS
ext4	3.5GB	16s	MD5 verified	PASS
ext4	20GB	110s	e2fsck clean	PASS

Additional mechanism tests:

• Idempotency: cryptsetup isLuks detects existing LUKS, reencrypt --encrypt rejects already-encrypted devices ("Device is already LUKS device. Aborting.")
• LUKS2 header: 16MB data offset with 32MB reserve (extra 16MB becomes usable space)
• 16MB reserve also works, 32MB used as safety margin per cryptsetup convention

Not tested (requires project CI)

• TPM2 auto-unlock on second boot (requires TPM2 enrollment during first-boot encryption, which needs the first-boot script running in a real initrd built by the project's RPM/image pipeline)
• Recovery key generation and use
• The first-boot script running automatically via the dracut module (tested manually; automatic execution requires the initramfs to be built by the project's standard image build process, not a Containerfile overlay)

Fixes: #2089
Related: #421, #476, #477

Signed-off-by: Andrew Dunn andrew@dunn.dev
AI-Assisted: yes
AI-Tools: GitLab Duo, OpenCode

AI-Generated Content Disclosure: This PR contains code generated with assistance from GitLab Duo and OpenCode. The output has been reviewed for correctness, tested, and validated against project requirements per GitLab's AI contribution guidelines.

[gemini-code-assist]

Code Review

This pull request introduces a new first-boot LUKS encryption mechanism for bootc. It defers the actual encryption and TPM2 enrollment of the root partition from installation time to the first boot, addressing issues with TPM2 enrollment in the install environment. This is achieved by reserving space for the LUKS header during installation, adding a new bootc-luks-firstboot.sh script and systemd service to the initrd, and updating the dracut module to include necessary components. Review comments highlight a critical flaw in the idempotency logic of the first-boot encryption script, which could leave the system unbootable if interrupted, and suggest a more robust method for parsing kernel command-line arguments.

[gemini-code-assist]

The script's idempotency logic is flawed. If the script is interrupted after encrypt_root but before configure_system completes, on the next boot should_encrypt will detect that the device is already a LUKS device and cause the script to exit. This will leave the system in an unbootable state because the bootloader configuration has not been updated to unlock the LUKS device.

The logic should be changed to ensure configure_system is always run if rd.bootc.luks.encrypt is present, even if the device is already encrypted. The should_encrypt function should be removed as its logic is now integrated into this main execution block.

Suggested change

	if ! should_encrypt; then
	log "No encryption requested or already encrypted. Exiting."
	exit 0
	fi
	encrypt_root
	configure_system
	if [ -z "$ENCRYPT_KARG" ]; then
	log "No encryption requested. Exiting."
	exit 0
	fi
	if [ -z "$ROOT_DEV" ]; then
	die "rd.bootc.luks.encrypt set but no root= device found"
	fi
	if ! cryptsetup isLuks "$ROOT_DEV" 2>/dev/null; then
	encrypt_root
	else
	log "Root device $ROOT_DEV is already LUKS. Skipping encryption."
	fi
	configure_system

[gemini-code-assist]

The current method of parsing /proc/cmdline using for arg in $cmdline relies on word splitting, which can fail if a kernel argument contains spaces. While it works for the current set of arguments, it's not robust. A safer approach is to read the command line into an array.

Suggested change

	local cmdline
	cmdline=$(< /proc/cmdline)
	for arg in $cmdline; do
	case "$arg" in
	rd.bootc.luks.encrypt=*)
	ENCRYPT_KARG="${arg#rd.bootc.luks.encrypt=}"
	;;
	root=UUID=*)
	local uuid="${arg#root=UUID=}"
	ROOT_DEV=$(blkid -U "$uuid" 2>/dev/null) || true
	;;
	root=/dev/*)
	ROOT_DEV="${arg#root=}"
	;;
	esac
	done
	local arg
	local -a cmdline_args
	read -r -a cmdline_args < /proc/cmdline
	for arg in "${cmdline_args[@]}"; do
	case "$arg" in
	rd.bootc.luks.encrypt=*)
	ENCRYPT_KARG="${arg#rd.bootc.luks.encrypt=}"
	;;
	root=UUID=*)
	local uuid="${arg#root=UUID=}"
	ROOT_DEV=$(blkid -U "$uuid" 2>/dev/null) || true
	;;
	root=/dev/*)
	ROOT_DEV="${arg#root=}"
	;;
	esac
	done