ZFS, LUKS and ZFS + LUKS might need additional clarification

[alecStewart1]

Apologies for the small blog post level of an issue ahead of time:

With installing Chimera, I've found myself digging around the internet as I think there's some missing information regarding
a LUKS + ZFS installation as well as a few other things in the documentation.

Say we want the following file system structure:

• EFI boot partition
  • This apparently needs to be at /boot/efi for update-initramfs script to work
    (specifying the boot directory doesn't seem to work)
• The LUKS encrypted partition (/dev/mapper/whatever)
  • A ZFS zpool (named zpool, rpool, whatever)
    • One mountpoint of rpool/ROOT/chimera at /
    • One mountpoint of /rpool/home at /home

One thing is that /boot/efi should only necessary if you're using something like GRUB. Just /efi is the "new"/"preferred" way
of doing things. I don't like GRUB, I just create EFI entries with efibootmgr. However, if update-initramfs specifically looks
for /bootthen I assume I can't just have a non-nested ESP partition mounted with /efi. When I chrooted into an installation that
was mounted this way, update-initramfs wouldn't put any kernel image or initramfs file in /efi, even if I specify the boot
directory. I feel like this shouldn't be the case, but it's entirely possible I messed something up or I'm missing something.

So we have our /dev/mapper/whatever LUKS device after running all of our cryptsetup commands.
We create our zpool:

zpool create -f -o ashift=12 \
  -o autotrim=on \
  -O compression=zstd \
  -O acltype=posixacl \
  -O atime=off \
  -O relatime=off \
  -O normalization=formD \
  -O dnodesize=auto \
  -O canmount=off \
  -O xattr=sa \
  -m none rpool /dev/mapper/whatever

There are a few things to note:

1. I don't know if normalization=formD is now standard or even necessary
2. autotrim=on might be useful for NVMe devices, but with our /etc/crypttab we'll have discard as well, so my
   initial assumption is that it could be redundant.
3. Having both relatime=off and atime=off might not be necessary, but I'm not sure.

These should likely be mentioned in the Root of ZFS section.

Void and other distros suggest exporting and then re-importing the pool with /mnt, in our case /media/root/.

zpool export rpool
zpool import -N -R /media/root rpool

Maybe this is more correct, maybe it doesn't matter.

Now we add our ZFS mount points for / and /home:

zfs create -o mountpoint=none rpool/ROOT
zfs create -o mountpoint=/ -o canmount=noauto rpool/ROOT/chimera
zfs create -o mountpoint=/home rpool/home

The Void Linux ZFS guide mentions having a cachefile: https://docs.voidlinux.org/installation/guides/zfs.html#mount-the-zfs-hierarchy

At this point, the entire ZFS hierarchy should be mounted and ready for installation. To improve boot-time import speed, it is useful
to record the current pool configuration in a cache file that Void will use to avoid walking the entire device hierarchy to identify
importable pools

With what Void's installation mentions, this is what it might look like for Chimera:

mkdir -p /media/root/etc/zfs
zpool set cachefile=/media/root/etc/zfs/zpool.cache rpool

Probably something worth mentioning.

We now mount the ESP partition, in this case (unless I'm totally wrong) it has to be /boot/efi:

mkdir -p /media/root/boot/efi
mount /dev/nvme0n1p1 /media/root/boot/efi

We bootstrap it, we chroot into it; lovely.

We have our /dev/sda2 or /dev/nvme0n1p2 that's the LUKS partition, and we have our /dev/mapper/whatever zpool.
To make our /etc/fstab we can run genfstab -t UUID / >> /etc/fstab.
For the /etc/crypttab we can do something like echo whatever UUID=XXXXX-XXXX-XXXX-XXXXX none luks,initramfs,discard > /etc/crypttab.

Now for the EFI entry. With GRUB, we would obiviously do grub-install --target=x86_64-efi --efi-directory=/boot/efi, and with just efibootmgr it's a whole thing. Maybe it's worth mentioning setups that don't use a bootloader.

We install linux-lts-zfs-bin or linux-stable-zfs-bin, all our other packages.

My question is:

What should root= be in the kernel commandline? The docs say:

You can freely combine ZFS and LUKS. Just keep in mind that when setting up root=, you do not have to care about any of the /dev/mapper
stuff, and simply specify the same root= as you would with an unencrypted system.

So does this mean root= should be the same UUID that's in /etc/crypttab, i.e. root=UUID=XXXXXXXX-XXXX-XXXX-XXXXXXX?

If the kernel needs to find a ZFS mount point, would we have to do root=ZFS=UUID-XXXXXXXX-XXXX-XXXX-XXXXXX (I don't think you can do that), or do we do
root=ZFS=rpool/ROOT/chimera to point it to the ZFS / mountpoint? I'm assuming the last of these.

EDIT: root=ZFS=rpool/ROOT/chimera seems to be correct looking around. It's also apparently worth sometime adding rootdelay= to your kernel commandline when booting with ZFS.

[alecStewart1]

Trying this again, it seems there's more details missing. One is that chimera-bootstrap needs to be run before

• adding a zpool cache file
• mounting the zfs datasets (zfs mount -a)

The other is that cryptsetup, or at least the version used by Chimera, might not support zfs as the underlying file system.

update-initramfs will come back with cryptsetup: WARNING: could not determine root device. If after running chimera-chroot and installing cryptsetup-scripts on the chrooted environment you need to run cryptsetup luksOpen ${device name} needs to be done again, even though it was open prior to running chimera-chroot in order to actually make it a ZFS filesystem, then that should probably be documented somewhere or there's something wrong with the version of cryptsetup-scripts or chimera-chroot.

I could be wrong here about a lot of things, but this is what my experience has been with installing Chimera given the documentation, but even then I had to have Alpine, Void, Gentoo and sometimes Arch's documentation up when figuring some of this stuff out.