ZFS 2.1.5_1 breaks boot on non zfs-on-root systems

[adrian-bl]

Is this a new report?

Yes

System Info

Void 5.18.4_1 x86_64 GenuineIntel uptodate rFFF

Package(s) Affected

zfs-2.1.5_1

Does a report exist for this bug with the project's home (upstream) and/or another distro?

No response

Expected behaviour

The system boots

Actual behaviour

After upgrading 2 systems from ZFS 2.1.4 to 2.1.5, both systems fail to boot as dracut complains about not being able to mount the rootfs.

Note that ZFS is not used as rootfs: one system uses ext4, the other btrfs.

Steps to reproduce

1. Have a non root-on-zfs system
2. Install zfs-2.1.5_1
3. Reboot -> system will fail to startup

[adrian-bl]

Digging around, the issue seems to be in zfs-lib.sh overwriting root with an old value.

I hacked up the following patch which fixes the issue for me:

--- zfs-lib.sh.orig	2022-06-24 12:46:00.250658147 +0200
+++ zfs-lib.sh	2022-06-24 12:39:42.823358632 +0200
@@ -87,11 +87,11 @@
         return
     fi
 
-    root=$(getarg root=)
+    xroot=$(getarg root=)
     rootfstype=$(getarg rootfstype=)
 
     # shellcheck disable=SC2249
-    case "$root" in
+    case "$xroot" in
         ""|zfs|zfs:|zfs:AUTO)
             root=zfs:AUTO
             rootfstype=zfs
@@ -99,7 +99,7 @@
             ;;
 
         ZFS=*|zfs:*)
-            root="${root#zfs:}"
+            root="${xroot#zfs:}"
             root="${root#ZFS=}"
             root=$(echo "$root" | tr '+' ' ')
             rootfstype=zfs

[zdykstra]

If you don't have root-on-ZFS, there's no need for zfs to be in your initramfs. You can add:

omit_dracutmodules+=" zfs " to /etc/dracut.conf.d/zfs.conf to exclude it completely on your next initramfs rebuild.

What kernel commandline do you boot your system with?

[ahesford]

The real question is why this never bit you before.

This doesn't really seem to be a bug; I suspect you aren't specifying any root= argument in your kernel command line, which means the ZFS module here assumes control and tries to auto-detect a pool. If you specify a root argument that doesn't specify ZFS stuff, this function should set root= to whatever the value is that you specified and leave it alone.

Also, as zdykstra mentioned, you shouldn't put the ZFS dracut module in your initramfs if you aren't booting a ZFS root.

[adrian-bl]

@zdykstra

What kernel commandline do you boot your system with?

[root@abyss adrian]# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-5.18.4_1 root=UUID=976f527a-5bf9-442c-be36-f57d98cd86d3 ro loglevel=4 rd.debug

@ahesford

No, i have root specified (see above)

Also, as zdykstra mentioned, you shouldn't put the ZFS dracut module in your initramfs if you aren't booting a
ZFS root.

Well, both systems are a pretty vanilla void installation - just with the ZFS package installed (because my /home is on ZFS). I don't think it is expected that i'd have to blocklist ZFS in dracut in such a configuration.

this function should set root= to whatever the value is that you specified and leave it alone.

The problem here is that dracut already changed this value before:

So my system sets root=UUID=976..., let's look at dracut's parse-block.sh:

case "$root" in
    block:LABEL=*|LABEL=*)
        root="${root#block:}"
        root="$(echo $root | sed 's,/,\\x2f,g')"
        root="block:/dev/disk/by-label/${root#LABEL=}"
        rootok=1 ;;
    block:UUID=*|UUID=*)
        root="${root#block:}"
        root="${root#UUID=}"
        root="$(echo $root | tr "[:upper:]" "[:lower:]")"
        root="block:/dev/disk/by-uuid/${root#UUID=}"
        rootok=1 ;;

Once this has run, root is set to block:/dev/disk/by-uid/976... (i verified that this is the case by booting with rd.debug)

Now later (after parse-block.sh but before the rootfs mount)zfs-lib.sh executes

root=$(getarg root=)

which will reset root with the value specified on the kernel command line, undoing whatever parse-block.sh did and ultimatively confusing the rest of dracut by having root set to a value it doesn't understand/expect.

(Also see here for the rd.debug output on how root is set/updated (note that this includes the patch i posted above (with xroot)) https://gist.githubusercontent.com/adrian-bl/49f25982d68b2e9ca4907b19ad4ce57b/raw/f72cbd3eb2d0fa461aa14279cf3b1fcb7c76866c/gistfile1.txt)

[ahesford]

Thanks for the clarification. This is most definitely a bug and should be fixed upstream (see openzfs/zfs#13589 for my proposal). Once we have some feedback from upstream, we can pull a patch back here. In the meantime, disabling the ZFS dracut module will get you what you need.

Also, cc: @Vaelatern

[Vaelatern]

The real question is why this never bit you before.

It wasn't there on an earlier release.

[Vaelatern]

My approach ^

[ahesford]

Should be fixed by d6d1cc4, sorry we didn't catch this before the update.

[adrian-bl]

Thanks! I can confirm that the updated package fixes the issue

[aaFn]

I got hit by that bug, and now my system doesn't boot anymore because of that, and drops to dracut shell.
How can I boot to a working state from dracut ? Is there a way to regenerate the initramfs from inside dracut, applying the patch by hand ??

[RunningDroid]

@aaFn You should be able to run lsblkid to locate your root partition and do mount $root /sysroot then exit or control-d to boot your system.

[aaFn]

Hello @RunningDroid , that worked, thank you (I was not aware that exiting the shell was continuing the process after that .. I learnt something today).