grub "root=ZFS=..." misses zpool

[heilts]

Hello,

I use Debian "Jessie" with zfs 0.6.4-1.1-2 and root on zfs. The root pool is "tank0".

After upgrading to grub 2.02-beta2.9-ZOL1, the grub.cfg misses the pool part, i.e. it becomes e.g.

linux /jessie-1/ROOT@/boot/vmlinuz-3.19.8 root=ZFS=jessie-1/ROOT ro boot=zfs $bootfs quiet

This does not work, it should be e.g.

linux /jessie-1/ROOT@/boot/vmlinuz-3.19.8 root=ZFS=tank0/jessie-1/ROOT ro boot=zfs $bootfs quiet

Also, at running "update-grub", I got

root@tor:~# update-grub
Generating grub configuration file ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-3.19.8
Found initrd image: /boot/initrd.img-3.19.8
/usr/sbin/grub-probe: error: unknown filesystem.
Found linux image: /boot/vmlinuz-3.19.3
Found initrd image: /boot/initrd.img-3.19.3
Found linux image: /boot/vmlinuz-3.16.0-4-amd64
Found initrd image: /boot/initrd.img-3.16.0-4-amd64
/usr/sbin/grub-probe: error: unknown filesystem.
Found memtest86+ image: /jessie-1/ROOT@/boot/memtest86+.bin
Found memtest86+ multiboot image: /jessie-1/ROOT@/boot/memtest86+_multiboot.bin
Found Debian GNU/Linux (8.0) on /dev/sdi1
done

The root filesystem is placed on GPT parted disks (raidz1) partitition 4. The other partititions are used for gtp grub and md1 (carrying an ext4 based rescue system and swap, this is why the update-grub creates so many entries). In more detail:
3.19.8 and 3.19.3 are one zfs (active root) only.
3.16.0-4-amd64 andn memtest are one zfs and ext4 (rescue).
And there is a complete backup of zfs placed on /dev/sdi1. This looks to break GRUB as well, because it creates only entries for 3.16.0-4-amd64, these multiple times, with increasing numbers of auf /dev/sdi1, looking like:

        menuentry 'Debian GNU/Linux (auf /dev/sdi1) (auf /dev/sdi1) (auf /dev/sdi1) (auf /dev/sdi1) (auf /dev/sdi1) (auf /dev/sdi1) (auf /dev/sdi1) (on /dev/sdi1)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/boot/vmlinuz-3.16.0-4-amd
64--51a2e90e-b2a5-4006-ba1b-db85588a6881' {
                insmod part_msdos
                insmod ext2
                set root='hd7,msdos1'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd8,msdos1 --hint-efi=hd8,msdos1 --hint-baremetal=ahci8,msdos1 --hint='hd7,msdos1'  51a2e90e-b2a5-4006-ba1b-db85588a6881
                else
                  search --no-floppy --fs-uuid --set=root 51a2e90e-b2a5-4006-ba1b-db85588a6881
                fi
                linux /boot/vmlinuz-3.16.0-4-amd64 root=UUID=8e3538db-de20-4ef4-81f2-9464fdeed165 ro quiet
                initrd /boot/initrd.img-3.16.0-4-amd64
        }

grub-probe works:

root@tor:~# grub-probe /
zfs

GRUB installation is:

root@tor:~# dpkg -l \*grub\* | grep ^ii
ii  grub-common     2.02-beta2.9-ZOL11-7aa9f6 amd64        GRand Unified Bootloader (common files)
ii  grub-pc         2.02-beta2.9-ZOL11-7aa9f6 amd64        GRand Unified Bootloader, version 2 (PC/BIOS version)
ii  grub-pc-bin     2.02-beta2.9-ZOL11-7aa9f6 amd64        GRand Unified Bootloader, version 2 (PC/BIOS binaries)
ii  grub2-common    2.02-beta2.9-ZOL11-7aa9f6 amd64        GRand Unified Bootloader (common files for version 2)

[heilts]

I tracked a bit further down:
In 10_linux the command line is build:

10_linux:       rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
10_linux:       LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs}"

But rpool becomes empty, because the grub_probe fails:

root@tor:/etc/grub.d# /usr/sbin/grub-probe --device /dev/sdb4 --target=fs_label
/usr/sbin/grub-probe: error: unknown filesystem.

Same happens with sdba4.

Note: My zfs root is placed on sda4 and sdb4 in raidz1.

[dechamps]

I got the exact same issue on Debian Sid, grub-common 2.02-beta2.9-ZOL11-7aa9f6. Using /, grub-probe works:

# grub-probe --target device /
/dev/sdd1
/dev/sdg1
/dev/sda1
/dev/sde1
/dev/sdc1
/dev/sdf1

However, when using grub-probe with a device node (which is what grub-mkconfig and 10_linux do), it doesn't:

# grub-probe --target fs --device /dev/sdd1
grub-probe: error: unknown filesystem.
$ sudo grub-probe --target fs_label --device /dev/sdd1
grub-probe: error: unknown filesystem.

I pasted -vvv output into https://gist.github.com/dechamps/8ebdf9c3631bd0cda2ef.

[dechamps]

For now, this can be worked around without touching package files by manually specifying root=ZFS=... in /etc/default/grub, in the GRUB_CMDLINE_LINUX variable. The contents of this variable are appended after the generated root stanza, which means the broken root will be overriden.

[firaxis]

I'm trying to install debian with zfs on root.
It installs and works with single vdev in pool or mirror, but not with raidz as boot pool.

[Spongman]

is this repo still active? i'm seeing exactly the same issue, 18 months later...

[Spongman]

apparently the bug is due to the fact that grub-mkconfig is doing a 'head -n1' on your grub-probe output. and later on when grub-probe is called again it needs to be passed a quorum of devices in order to determine the filesystem type and label.

[firaxis]

grub-probe can't determine filesystem type when you are trying to get it from one of raidz disks.
Also it may depend on array size, e.g. grub-probe can't do it on array of > 6 disks.

[pstch]

This can be worked around by editing /etc/grub.d/10_linux and replacing:

rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`

By :

rpool=`zdb -l ${GRUB_DEVICE} | grep " name:" | cut -d\' -f2`

You can also just hardcode your root pool name:

rpool="rpool"

[exikyut]

Hello everyone, I believe I've just crashed into exactly this issue, an extremely similar issue, or the same outcome caused by different circumstances.

In my case, grub-probe is not detecting the root pool due to unsupported features. It is incidentally actually able to detect an auxillary pool though:

# grub-probe -t fs_label -d /dev/vda3
grub-probe: error: unknown filesystem.
# grub-probe -t fs_label -d /dev/vda4
pool-1

Adding -vvvv reveals that it bails out during feature detection:

grub-core/fs/zfs/zfs.c:2115: zap: name = org.illumos:lz4_compress, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.joyent:multi_vdev_crash_dump, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:hole_birth, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:extensible_dataset, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:embedded_data, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.open-zfs:large_blocks, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.zfsonlinux:large_dnode, value = 1, cd = 0
grub-core/kern/fs.c:78: zfs detection failed.

versus

grub-core/fs/zfs/zfs.c:2115: zap: name = org.illumos:lz4_compress, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.joyent:multi_vdev_crash_dump, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:hole_birth, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:extensible_dataset, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:embedded_data, value = 1, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.open-zfs:large_blocks, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.zfsonlinux:large_dnode, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.illumos:sha512, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.illumos:skein, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = org.illumos:edonr, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.datto:bookmark_v2, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.datto:encryption, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = com.delphix:device_removal, value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = , value = 0, cd = 0
grub-core/fs/zfs/zfs.c:2115: zap: name = , value = 0, cd = 0
pool-1
grub-core/kern/disk.c:295: Closing `hostdisk//dev/vda'.

Note how large_dnode is 1 in the failing case; this value is not in the whitelist.

I make the (IMO reasonably solid) argument over at https://savannah.gnu.org/bugs/index.php?58555 that this is because grub-probe is brokenly being used to analyze root filesystems with features enabled that it does not support.

If I remember :) I'll update this thread.

[jstrieter]

This can be worked around by editing /etc/grub.d/10_linux and replacing:

rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`

By :

rpool=`zdb -l ${GRUB_DEVICE} | grep " name:" | cut -d\' -f2`

You can also just hardcode your root pool name:

rpool="rpool"

This workaround fixes it but it's still an issue as of 2.05_alpha20200310.

[crocket]

This is an issue on my machine with grub-2.05_alpha20200310

[MilesBHuff]

What's the long wait on this issue for? It's been open for 6 years. There's even a fix in the comments.

[jypma]

Apparently a patch was submitted to grub itself back in december, which was reviewed but then got dropped due to some other side-effect (see replies in that thread). I wonder if it just needs the final nudge.

[vhns]

Apparently a patch was submitted to grub itself back in december, which was reviewed but then got dropped due to some other side-effect (see replies in that thread). I wonder if it just needs the final nudge.

It's because that patch throws a grub_probe error. Here's the output I get upon using GRUB patched with it:

vhns@vhns ~ $ grep -i rpool -A 5 /boot/grub/grub.cfg
        linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop root=ZFS=rpool/ROOT/gentoo ro  
        echo    'Loading initial ramdisk ...'
        initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
}
submenu 'Advanced options for Gentoo GNU/Linux' $menuentry_id_option 'gnulinux-advanced-/dev/sdb4' {
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop-advanced-/dev/sdb4' {
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop root=ZFS=rpool/ROOT/gentoo ro  
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop (recovery mode)' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop-recovery-/dev/sdb4' {
                load_video
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop root=ZFS=rpool/ROOT/gentoo ro single 
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop.old' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop.old-advanced-/dev/sdb4' {
                load_video
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop.old root=ZFS=rpool/ROOT/gentoo ro  
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop.old (recovery mode)' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop.old-recovery-/dev/sdb4' {
                load_video
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop.old root=ZFS=rpool/ROOT/gentoo ro single 
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
}

vhns@vhns ~ $

And here's with @pstch 's patch:

vhns@vhns ~ $ sudo grub-mkconfig -o /boot/grub/grub.cfg 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.10.52-gentoovhns-work-laptop
Found initrd image: /boot/intel-uc.img /boot/initramfs-5.10.52-gentoovhns-work-laptop.img
Found linux image: /boot/vmlinuz-5.10.52-gentoovhns-work-laptop.old
Found initrd image: /boot/intel-uc.img /boot/initramfs-5.10.52-gentoovhns-work-laptop.img
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done
vhns@vhns ~ $ grep -i rpool -A 5 /boot/grub/grub.cfg
        linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop root=ZFS=rpool/ROOT/gentoo ro  
        echo    'Loading initial ramdisk ...'
        initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
}
submenu 'Advanced options for Gentoo GNU/Linux' $menuentry_id_option 'gnulinux-advanced-/dev/sdb4' {
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop-advanced-/dev/sdb4' {
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop root=ZFS=rpool/ROOT/gentoo ro  
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop (recovery mode)' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop-recovery-/dev/sdb4' {
                load_video
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop root=ZFS=rpool/ROOT/gentoo ro single 
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop.old' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop.old-advanced-/dev/sdb4' {
                load_video
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop.old root=ZFS=rpool/ROOT/gentoo ro  
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
        menuentry 'Gentoo GNU/Linux, with Linux 5.10.52-gentoovhns-work-laptop.old (recovery mode)' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.10.52-gentoovhns-work-laptop.old-recovery-/dev/sdb4' {
                load_video
--
                linux   /BOOT/gentoo@/vmlinuz-5.10.52-gentoovhns-work-laptop.old root=ZFS=rpool/ROOT/gentoo ro single 
                echo    'Loading initial ramdisk ...'
                initrd  /BOOT/gentoo@/intel-uc.img /BOOT/gentoo@/initramfs-5.10.52-gentoovhns-work-laptop.img
        }
}

vhns@vhns ~ $ 

[pstch]

@vhns Your two pastes are identical.

[tomtastic]

I got bitten by this recently, pool name not being included in the line of linux kernel options :<
I've followed up with Daniel Kiper from the grub mailing list thread, I'll try to get him to reconsider the patch from Doug Goldstein and get this bug fixed finally.

[RyukaZou]

Hi, this issue seem still exist in grub 2.06-r2. I recently try to install gentoo on zfs, after all installation process and out of chroot everything work, but when I rebuild the system with emerge -e @world and after it is done, I do grub-mkconfig -o /boot/grub/grub.cfg then system become unbootable.

I will try the work around in this issue and update the result.

[AkechiShiro]

@tomtastic is there any news on this patch landing so this annoying bug can be fixed soon ?

[tomtastic]

No sadly Daniel Kiper ignored my message to the mailing list and also those I sent him individually.
https://www.mail-archive.com/grub-devel@gnu.org/msg33490.html

[IvanVolosyuk]

It is kinda said that the bug is not being fixed. I don't have bpool and rpool split and grub happily leaves in uefi vfat32 partition, while root is an unsupported by grub ZFS filesystem (with feature flags like compression, etc). The grub-probe doesn't recognize it as ZFS.

I replaced rpool= in 10_linux with:

rpool=`blkid $GRUB_DEVICE --output export|grep LABEL|cut -d= -f2|head -n1`

[amano-kenji]

Maybe, it's time to replace grub with something else.

[IvanVolosyuk]

Grub gets the work done and has low complexity and sufficient flexibility for my needs.

[amano-kenji]

Grub itself doesn't have low complexity, but the fact that people stopped caring about fixing grub bugs is enough for me to look for alternatives.

Someone already proposed a bug fix which has been ignored for years. This is a serious issue in itself.