Cannot import pool with one device forced offline

[lanodan]

System information

Type	Version/Name
Distribution Name	Gentoo Linux
Distribution Version	N/A
Linux Kernel	5.4.20-gentoo
Architecture	amd64
ZFS Version	zfs-0.8.3-r1-gentoo zfs-kmod-0.8.3-r0-gentoo

Hello,

I wanted to add a new disk to my mirror because of disks potentially becoming faulty. Problem is that I never remember the difference between zpool add and zpool attach (I think an alias of zpool attach to zpool vdev-add would be useful to me) so I ended up doing a zpool add zroot root3.

I saw that I had it wrong immediately in zpool status so I wanted to remove it, done a zpool remove zroot root3, which greated me with something like "Can't remove device in root pool because GRUB", okay wtf GRUB should be fixed then, cannot be forced so I then did a zpool offline zpool root3 which didn't work (IIRC something about replicas) so I did a zpool offline -f zroot root3.

Now, I rebooted and ended up with the following (eye-copied from TTY so formatting is probably slightly wrong):

rescueshell / # zpool import
   pool: zroot
 state: UNAVAIL
status: One or more devices are faulted.
action: The pool cannot be imported due to damaged devices or data.
config:

    zroot         UNAVAIL  insufficient replicas
      mirror-0  ONLINE
        root1     ONLINE
        root2     ONLINE
     root3        FAULTED  corrupted data
rescueshell / # zpool import zroot
cannot import 'zroot': one or more devices is currently unavailable

Options like -F or -F -X doesn't seems to be able to rescue it, thing is: root3 basically contains useless data if any. Meanwhile zroot is quite important, I can restore it but that's painful to do so because I would have to reinstall everything and loose the data that wasn't in the last backup. Will do an install on another disk, hopefully a temporary one.

Lesson learned there: double-check the manpage, maybe try on files first, abuse of -n (which should be added to zpool attach, right?).

[rincebrain]

You could turn up zfs_max_missing_tvds to 1 and try importing with that to recover your data.

As far as GRUB incompatibilities, that's one reason most guides recommend keeping rootfs and bootfs in separate pools.

[lanodan]

[2020-04-25 22:17:58-0700] Rich Ercolani:

You could turn up zfs_max_missing_tvds to 1 and try importing with that to recover your data.

Thanks a lot! Currently copying the zfs filesystems to a new pool and will check it's data once that's done.

As far as GRUB incompatibilities, that's one reason most guides recommend keeping rootfs and bootfs in separate pools.

Yet to see that in Linux guides (other than in a way Alpine wiki still putting ext* for the /boot) but it does sounds like a good idea (which should be justified as otherwise I'm pretty sure I would have ignored it). And I guess it's a little bit irrelevant in FreeBSD/Illumos as they don't use grub by default but could still be done in their installer I guess.