Unable to set bootfs on pool with indirect vdevs

[mmatuska]

System information

All ZFS versions since device removal feature on both FreeBSD and Linux

Describe the problem you're observing

When a vdev is removed and becomes indirect, the bootfs property cannot be set on a pool.
I do not see the point why setting this property should not be allowed.
FreeBSD boot loader is capable of booting from indirect vdevs so this kills the management of bootability of the pool on FreeBSD.

This limitation was introduced in a1d477c by @ahrens

Describe how to reproduce the problem

dd if=/dev/zero of=/tmp/file1 bs=1M count=512
dd if=/dev/zero of=/tmp/file2 bs=1M count=512
zpool create testpool /tmp/file1 /tmp/file2
zpool remove testpool /tmp/file2
zpool set bootfs=testpool testpool

Include any warning/errors/backtraces from the system logs

cannot set property for 'testpool': operation not supported on this type of pool

References

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254349

Original reporter

Eric van Gyzen @vangyzen

[mmatuska]

@tsoome could you please tell if our loader officially supports booting on pools with removed vdevs? It looks like you have been handling VDEV_TYPE_INDIRECT code in there: freebsd/freebsd-src@b1b9326

If not, this can become dangerous for people removing vdevs from their boot pools.

[tsoome]

@tsoome could you please tell if our loader officially supports booting on pools with removed vdevs? It looks like you have been handling VDEV_TYPE_INDIRECT code in there: freebsd/freebsd-src@b1b9326

If not, this can become dangerous for people removing vdevs from their boot pools.

pools with removed vdevs should be fine. at least last time when I did test, it was ok:)

[vangyzen]

My workstation, running FreeBSD main from 2020-12-25, boots from such a pool. That's how I found the issue. My old SSD was failing, so I migrated to a new one. For historical reasons, my pool was split across two partitions on the old disk. I wanted to use only one on the new disk, so I didn't use mirroring for the migration. I simply added and removed vdevs.

[nabijaczleweli]

Closable? Seems to've landed in #11763

Fixed by #11763