Regression between 101 and 102: LUKS root filesystem is broken

[LaserEyess]

Describe the bug
When upgrading to 102, and after adding systemd-cryptsetup as a module, automatic decryption with a tpm2 locked root drive no longer works.

Distribution used
Arch Linux

Dracut version
102

Init system
systemd

To Reproduce
Use systemd-cryptsetup to bind a LUKS root partition to a TPM device, then add

add_dracutmodules=" tpm2-tss systemd-cryptsetup "

To your config along with the corresponding commandline options:

root=/dev/mapper/root rw rd.luks.name=$UUID=root rd.luks.options=no-read-workqueue,no-write-workqueue,tpm2-device=auto

Expected behavior
Auto decyption of root if tpm pcr conditions are met

Downgrading to dracut 101 works.
Additional context
Logs of the failed boot on 102 (truncated):

May 31 18:09:12 charlotte systemd[1]: Starting Cryptography Setup for luks-2f3c07db-91ed-4233-bd13-82a37b8996e0...
May 31 18:09:12 charlotte systemd-cryptsetup[648]: Encountered unknown /etc/crypttab option '-', ignoring.
May 31 18:09:13 charlotte kernel: BTRFS: device label rootfs devid 1 transid 1462895 /dev/dm-0 scanned by (udev-worker) (425)
May 31 18:09:13 charlotte systemd[1]: Found device /dev/mapper/root.
May 31 18:09:13 charlotte systemd[1]: Reached target Initrd Root Device.
May 31 18:09:13 charlotte systemd[1]: systemd-cryptsetup@root.service: Deactivated successfully.
May 31 18:09:13 charlotte systemd[1]: Started systemd-cryptsetup@root.service.
May 31 18:09:14 charlotte systemd-cryptsetup[648]: Cannot use device /dev/nvme0n1p2 which is in use (already mapped or mounted).
May 31 18:09:14 charlotte systemd[1]: Started Dispatch Password Requests to Console.
May 31 18:09:14 charlotte systemd-tty-ask-password-agent[685]: Starting password query on /dev/tty1.

Also I have a remote shell and logging in shows /dev/mapper/root mounted properly. Entering the password to decrypt works but it gives these errors:

May 31 18:12:33 charlotte systemd-tty-ask-password-agent[685]: Starting password query on /dev/tty1.
May 31 18:12:49 charlotte systemd-cryptsetup[648]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/nvme0n1p2.
May 31 18:12:52 charlotte systemd-cryptsetup[648]: Cannot use device /dev/nvme0n1p2 which is in use (already mapped or mounted).
May 31 18:12:52 charlotte systemd-cryptsetup[648]: Failed to activate with specified passphrase: Device or resource busy
May 31 18:12:52 charlotte systemd[1]: systemd-cryptsetup@luks\x2d2f3c07db\x2d91ed\x2d4233\x2dbd13\x2d82a37b8996e0.service: Main process exited, code=exited, status=1/FAILURE
May 31 18:12:52 charlotte systemd[1]: systemd-cryptsetup@luks\x2d2f3c07db\x2d91ed\x2d4233\x2dbd13\x2d82a37b8996e0.service: Failed with result 'exit-code'.
May 31 18:12:52 charlotte systemd[1]: Failed to start Cryptography Setup for luks-2f3c07db-91ed-4233-bd13-82a37b8996e0.
May 31 18:12:52 charlotte systemd[1]: Dependency failed for Local Encrypted Volumes.
May 31 18:12:52 charlotte systemd[1]: cryptsetup.target: Job cryptsetup.target/start failed with result 'dependency'.
May 31 18:12:52 charlotte dracut-initqueue[647]: A dependency job for cryptsetup.target failed. See 'journalctl -xe' for details.
May 31 18:12:52 charlotte systemd[1]: systemd-cryptsetup@luks\x2d2f3c07db\x2d91ed\x2d4233\x2dbd13\x2d82a37b8996e0.service: Consumed 15.623s CPU time.

Logs from a successful boot on 101

Jun 03 16:09:06 charlotte systemd[1]: Starting Cryptography Setup for root...
Jun 03 16:09:08 charlotte kernel: BTRFS: device label rootfs devid 1 transid 1470172 /dev/dm-0 scanned by (udev-worker) (439)
Jun 03 16:09:08 charlotte systemd[1]: Found device /dev/mapper/root.
Jun 03 16:09:08 charlotte systemd[1]: Finished Cryptography Setup for root.

Please let me know what other information is helpful here. I suspect the generation of the /etc/crypttab got broken some how, but I'm not sure how.

[paulo-erichsen]

I've ran into a similar issue on Arch Linux. Posted on the Arch Linux forums similar logs with error messages. Not sure if the issue is with how Arch Linux packaged dracut-102 or if it's an upstream issue.

• https://bbs.archlinux.org/viewtopic.php?pid=2176404 - dracut 102: having cryptsetup issues lately

[berndf]

Probably the same issue here, on Gentoo... Solved by downgrading to sys-kernel/dracut-101.

dracut-102 starts systemd-cryptsetup@usbBoot.service on the encrypted device "usbBoot", trying to read a passphrase from /dev/tty1. Thankfully, after three failed attempts at that (if one enters three CRs...), processing continues as usual and dracut successfully employs rd.luks.name and rd.luks.key as instructed from the (grub) kernel command line.

BTW, the message "Encountered unknown /etc/crypttab option '-', ignoring." helped me find the archlinux report (thanks!) - the initrd carries no /etc/crypttab at all. Is that (wrongly) generated by the dracut hooks?

[LaszloGombos]

The area of code changed quite a bit upstream. It is not obvious how to reproduce it and and it not obvious which commit caused the regression, so if someone who is able to reproduce could help to bisect which git commit caused the issued that would be helpful.

I would start checking if perhaps 2339acf is the cause of the regression.

Would manually setting rd.auto=0 in the kernel command line works around the problem ?

CC @DanWin @adrelanos @pvalena

[LaserEyess]

Setting rd.auto=0 appears to have worked.

[LaszloGombos]

I also expect that there is no issue/regression with --hostonly (and without forcing rd.auto=0). Perhaps someone can help to confirm.

[LaserEyess]

I use hostonly="yes" but also hostonly_cmdline="no" in my config. The docs are unclear about how hostonly_cmdline and kernel_cmdline interact so I wanted to be explicit. WIth those settings, I have to specify rd.auto=0.

Edit: It looks like --hostonly by itself works, confirming your theory.

[LaszloGombos]

@LaserEyess Thanks. Would you be able to post your kernel command line after a successful boot here to better understand the scope of the problem ?

[LaserEyess]

From journalctl -b0:

Command line: root=/dev/mapper/root rw rd.luks.name=2f3c07db-91ed-4233-bd13-82a37b8996e0=root rd.luks.options=no-read-workqueue,no-write-workqueue,tpm2-device=auto amd_iommu=on

Same as what's in dracut.conf

[LaszloGombos]

@LaserEyess Thanks.

It seems hostonly_cmdline="no" would prevent reading kernel command line from dracut.conf.

Can you confirm that changing hostonly_cmdline="no" to hostonly_cmdline="yes" (or simply removing setting hostonly_cmdline as the default is yes in this case) would also resolve this issue for you ?

The docs are unclear about how hostonly_cmdline and kernel_cmdline interact

I agree with the lack of documentation here. The code seems suggest that hostonly_cmdline="no" would imply that command line needs to be passed in - e.g. from bootloader.

[LaserEyess]

Yes that's basically what I confirmed above, to summarize:

# Broken
hostonly="yes"
hostonly_cmdline="no"
kernel_cmdline="......"

# Working
hostonly="yes"
hostonly_cmdline="no"
kernel_cmdline="...... rd.auto=0"

# Working
hostonly="yes"
#hostonly_cmdline="no"
kernel_cmdline="......"

I think my reading of the docs at the time led me to believe this option would somehow override kernel_cmdline, which does not seem to be the case. Still, this behavior doesn't exactly explain why setting hostonly_cmdline breaks decryption for me.

[berndf]

I can confirm for my (gentoo) case that setting rd.auto=0 works to avoid the described deviation where the system first tries to read a passphrase from the terminal.

Note that I'm quite sure that all of this has nothing to do with tpm unlock specifically - happens in my case for simply using rd.luks.key.

[LaszloGombos]

Unless there is a better suggestion, I am leaning towards reverting 2339acf, but I'd like to give more time for discussion/consideration.

Revert would re-open dracutdevs/dracut#2437. It would be desired to find a (different) solution to the original problem without creating surprise regressions.

CC @Alphix