ZFS features support

[mmatuska]

Illumos has recently introduced a new concept called ZFS features support. This way new functionality can be added to ZFS pools independently providing or not providing backward compatibility. I am integrating this support into FreeBSD.

You can read more about the concept here:
http://lists.freebsd.org/pipermail/freebsd-fs/2011-May/011568.html

Illumos changesets:
https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/2889e2596bd6
https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/1949b688d5fb

[behlendorf]

@mmatuska Thanks for the references, yes I've been following the feature flag work done by Delphix for a while now. We absolutely plan to integrate those changes since we already have a few changes in out tree which probably warrant the use of a feature flag.

[Vilbrekin]

Hi,
I do second this feature request, as it's currently impossible to import pools created on the latest version of OI.
Rgds,
V

[behlendorf]

Presumably the OI pools fail to import because they claim zpool/zfs version 1000?

[Vilbrekin]

Yep, 5000 if I'm correct. Would be really nice to have a linux alternative.

[behlendorf]

Yup, we'll get there. In the meanwhile I'd suggest explicitly creating pools with version 28 if you want to use them on Linux. Incidentally, your OI pools may never work on Solaris since they might not adopt any of the feature flag changes.

[mmatuska]

Pull request #908 is one of the last milestones before feature flags support.
With that code committed, the feature flag diffs have only very few conflicts.

[behlendorf]

@mmatuska Alright, #908 has been merged

[csiden]

You probably also eventually want:
https://hg.openindiana.org/upstream/illumos/illumos-gate/rev/2aba784c276b

Without it you need to manually enable individual features through their properties, zpool upgrade won't do it. It also adds more detail to the zpool status and zpool import output.

[dechamps]

This is blocking #939.

[ryao]

@dechamps It doesn't appear that this is blocking #939. It is true that the diffs themselves depend on this, but nothing prevents them from being reworked to permit their inclusion ahead of this.

@behlendorf Will pool version 5000 be used by default when creating new pools or will users be required to explicitly request feature flag support as we do with the xattr extension? As you stated, this will break Solaris compatibility. My plan for Gentoo is to recommend the use of pool version 28 unless people have an explicit need for extensions.

[behlendorf]

@ryao My inclination is to go slow initially and leave the default pool version at 28. As we gain more confidence in the implementation we can consider making it the default. We're also going to need to update grub to understand feature flags.

[ryao]

Illumos GRUB already supports feature flags. The ZFS code in Illumos GRUB and GRUB2 is fairly similar, so it should be possible to port feature flag support from Illumos GRUB to GRUB2.

I do not plan to be the one to do that port. Most Gentoo Linux users prefer GRUB 0.97 while most Gentoo FreeBSD users prefer FreeBSD's bootloader. Anything that I do in the area of bootloaders is generally focused on one of those two. I am gradually developing port of Illumos GRUB to Gentoo Linux in ryao/grub-illumos. After that is done, I want to modify FreeBSD's bootloader to boot Gentoo Linux.

[lundman]

I did hear a rumour that FBSD's grub can also boot more advanced vdev setups. Ie, not only single and mirrors, but raidz? That would be desirable to port as well.

Has IllumOS discussed more broadly how the feature-flags would work, for example with possible future crypto support? Their own specs say all features should be enabled by default, which seems undesirable in this (and possibly other) case. Especially since encryption is optional on each ZFS "filesystem" and might not be in use at all in the pool. But having it enabled stops importing pools on other systems.

[csiden]

@behlendorf @ryao Is the intention to eventually switch to enabling all features by default (on new pools) once you're more comfortable with feature flags? The implementation in illumos is the way it is because the average user won't want to worry about every available feature flag, these won't all be exciting things like encryption support or new compression algorithms, for example feature@empty_bpobj is just a minor modification to the way we create snapshots that wastes less space/memory. Ideally we wouldn't expose feature@empty_bpobj to the user at all, but we need to in case the user cares about the portability of the pool. The average user though won't care if their pool will work on FreeBSD or illumos and just wants the newest functionality to work for them. The one piece of compatibility such a user would care about is "I just upgraded to a newer Linux/Gentoo/ZFS release and it's not working for me, I want to go back to the older release that doesn't support all the same ZFS feature flags", but that case is taken care of since new features are not automatically enabled during software upgrades. The user has to run 'zpool upgrade mypool' to enable all new features (but the user doesn't need to worry what the features are), which they can do once they are satisfied they won't want to downgrade.

@lundman I wrote this mostly in response to your question, although your answer got buried near the bottom:

Feature flags are meant as a replacement for the old zpool disk-format version number (the one that went from 1-28 and Oracle has continued in to the 30's) and should work in a very similar way. So for comparison:

With the version number, when new pools are created they automatically have the latest supported version number, unless the user specifies a lower version (zpool create -o version=20 ...). With feature flags, when a new pool is created all features are enabled, unless a pre-feature flags version number is selected (zpool create -o version=20 ...) or the feature is explicitly marked as disabled (zpool create -o feature@async_destroy=disabled ...).

With the version number, If a pool was created with an older software version and the software has since been upgraded to support a newer version number, the pool still retains the old version number and attempts to use features of ZFS on a pool whose on-disk version does not support them results in some ENOTSUP-like error. For example trying to do:

$ zpool create -o version=20 mypool
$ zfs create -o dedup=on mypool/myfs

You get an error:

cannot create 'mypool/myfs': pool must be upgraded to set this property or value

Because, while the software supports dedup, we could not write the dedup table to disk and still be compatible with on-disk versions prior to 21. Similarly with feature flags, if you upgrade the software to support a new feature that was not supported when a pool was first created that feature will remain disabled. If encryption were implemented in the future it would be done similarly to dedup, there would be a pool-wide feature flag to indicate that the on-disk format has changed, and a per-filesystem property to actually enable encryption on data, if you tried to set the filesystem property before the feature flag was enabled on the pool you would get a similar error to the one we got above for dedup.

Total side note:
There are also some big differences between the old version number and feature flags (other than the obvious one were version numbers imply a dependency between on-disk format changes even if there isn't one, while feature flags allows on-disk changes to be developed, ported, and enabled independently of each other). Mostly the differences deal with making feature flags more flexible than version numbers, pretty much everything described in https://github.com/illumos/illumos-gate/blob/master/usr/src/man/man5/zpool-features.5 isn't supported by version numbers.