Metadata Allocation Class

[don-brady]

Intel is working on ways to isolate large block file data from metadata for ZFS on Linux. In addition to the size discrepancy with file data, metadata often has a more transient lifecycle and additional redundancy requirements (ditto blocks). Metadata is often a poor match for a RAIDZ tier since it cannot be dispersed and the relative parity overhead is high. A mirrored redundancy is a better choice for metadata.

A metadata-only allocation tier is being added to the existing storage pool allocation class mechanism and is used as the primary source for metadata allocations. File data remains in the normal class. Each top-level metadata VDEV is tagged as belonging to the metadata allocation class and at runtime becomes associated with a pool’s metadata allocation class. The remaining (i.e. non-designated) top-level VDEVs default to the normal allocation class distinction. In addition to generic metadata, the performance sensitive deduplication table (DDT) data can also benefit from having its own separate allocation class.

Allocation Class	Purpose
Normal	Default source for all allocations
Log	ZIL records only
Metadata (new)	All metadata blocks (non level-0 file blocks)
Dedup Table (new)	Deduplication Table Data (DDT)

More details to follow.

[DeHackEd]

I swear someone else was working on a metadata-specific vdev for much the same purpose.

[grahamperrin]

@DeHackEd I have seen writings about the concept but FWIW, no recollection of anyone else working on it in an open source area.

Found today: #1071 (comment) mentioning tegile, and Google finds e.g.:

• http://www.tegile.com/blog/hybrid-unified-optimized-storage/ "… To the ZFS core they add a sophisticated in-line deduplication and meta-data acceleration code that takes advantage of the variety of storage types …"
• http://www.tegile.com/blog/george_tintri/ "… we uniquely leverage flash as a store for all metadata …"
• http://www.tegile.com/products/intelliflash

[nwf]

It might be nice to allow (optionally!) one (or two?) of the metadata ditto blocks to reside in the ordinary pool, as well, making the metadata vdevs a kind of dedicated, write-through cache. (Different from L2ARC with secondarycache=metadata because they really would be holding authoritative copies of the metadata, but in dire straits could still be removed from the pool or used at lower fault tolerance -- one SSD instead of two in a mirror, etc.)

[lschweiss-wustl]

I just watched the Livestream presentation on this. This is definitely a feature needed in ZFS. I've struggled to keep metadata cached. The only way I've been able to get decent amount of metadata cached is to set L2ARC to metadata only, but it takes many passes, which still probably misses a significant amount. I would love to build pools with metadata on SSD tiers. Metadata typically accounts for 1/2 my disk I/O load.

There was a presentation from Nexenta a couple years back about tiered storage pools that had similar goals. http://www.open-zfs.org/w/images/7/71/ZFS_tiering.pdf I haven't heard anything of this effort since.

[tuxoko]

Hi @don-brady
May I ask what's the current status of this feature? Is there anything I can help work on with?
Thanks.

[don-brady]

@tuxoko I'm hoping to post a public WIP branch soon. The creation/addition of VDEVs dedicated to specific metadata classes is functional. Currently working out accounting issues in metaslab layer. We just started running ztest with metadata only classes to help shake out any edge cases (found a few). Let me come up with a to-do list so others can help.

[tuxoko]

Sounds great!!

[inkdot7]

Hi @don-brady
Did some more work on and measurements for the #4365 implementation. Would be interesting to test also this version.

[adilger]

It seems that the WIP pull request #5182 was never referenced here, or vice versa...