What does the ZFS Metadata Special Device do?

[yuanfangcc]

zpool create pool sdb special sdc
This command create a pool with a metadata device sdc. I want to know what role with this device sdc.Under what circumstances will it be used?

[robn]

There's two types of "data" in the pool: the actual data of whatever it is you're storing, and the metadata, which is all the tables, properties, indexes and other "stuff" that defines the pool structure, the datasets, and the pointers that tell ZFS where on disk to find the actual data.

Normally this is all mixed in together on the regular pool vdevs (mirror, raidz, etc). If you add a special vdev to your pool, then ZFS will prefer to store the metadata there, and send the data proper to the regular vdevs. The main reason for doing this is if you have "slow" data vdevs; adding a special vdev of a SSD mirror can speed up access times, as ZFS can look to the SSDs to know where on the data vdevs to find its data and go directly there, rather than loading the metadata off the slow vdevs and then needing another access to get the real data.

There's another possible advantage: ZFS can store "small" files on the special vdev, leaving larger ones for the regular data vdevs.

One important thing to remember is that special vdevs are a proper part of the pool, not an add-on - just like the regular data vdevs, if the special vdev fails, the pool is lost. An SSD mirror is typical for this vdev.

This is only a rough explanation, see also:

• zpoolconcepts(7)
• Level1Techs writeup

[robn]

That's right!

[yuanfangcc]

ok,thanks

[yuanfangcc]

Excuse me again, adding special vdev is supposed to improve file system IOPS dramatically, but in my vdbench testing, I didn't see much of a performance improvement. What's the reason for this？
My special vdev used the SSD, other vdev used the HDD.

[MrRinkana]

One possible explanation for no change in performance is that adding a special device will not move existing metadata to it, only newly written. This could probably worked around by send/receiving the data (effectively rewriting everything)..

Edit: although I now realise that vbench is probably writing new data while testing, so my comment probably does not apply.

[amotin]

I don't know what vdbench was measuring exactly here, but: ZFS ARC happily caches the most actively accessed blocks, and metadata are often the such ones; for sequential I/O ZFS has speculative prefetcher, that is also able to to prefetch metadata. You'd have to beat those two heavily to get performance improvement out of special vdev. If vdbench writes and then reads the same data within one test, then metadata have very high change to appear in ARC. You may try to write something, then export/import the pool to wipe ARC, and then do some random read, then may be for some time untill ARC warm up you'll see the difference. Another option is a pool with really huge amount of metadata: tons of small files, directories, special attributes, ACLs, etc. -- those can't be prefetched well, and their size can be huge for ARC. Dedup table can be huge too, but that should better fit in ARC any way if you wish reasonable write performance.

[joshenders]

This page has a high rank for "openzfs metadata special" so hope this helps someone:

On Apr 24th, 2023, a Metadata Total line was added to zdb -bb output. No need to calculate it manually for metadata special device capacity planning, just reference the ASIZE column and add a little extra for future growth.

This commit is in >= zfs-2.2.0

commit 6b6aaf6dc2e65c63c74fbd7840c14627e9a91ce2
Author: Rich Ercolani <rincebrain@gmail.com>
Date:   Mon Apr 24 19:55:07 2023 -0400

    Taught zdb -bb to print metadata totals
    
    People often want estimates of how much of their pool is occupied
    by metadata, but they end up using lots of text processing on zdb's
    output to get it.
    
    So let's just...provide it for them.
    
    Now, zdb -bbbs will output something like:
    
    Blocks  LSIZE   PSIZE   ASIZE     avg    comp   %Total  Type
    [...]
        68  1.06M    272K    544K      8K    4.00     0.00      L6 Total
     1.71K   212M   6.85M   13.7M      8K   30.91     0.00      L5 Total
     1.71K   212M   6.85M   13.7M      8K   30.91     0.00      L4 Total
     1.73K   214M   6.92M   13.8M      8K   30.89     0.00      L3 Total
     18.7K  2.29G    111M    221M   11.8K   21.19     0.00      L2 Total
     3.56M   454G   28.4G   56.9G   16.0K   15.97     0.19      L1 Total
      308M  36.8T   28.2T   28.6T   95.1K    1.30    99.80      L0 Total
      311M  37.3T   28.3T   28.6T   94.2K    1.32   100.00  Total
     50.4M   774G    113G    291G   5.77K    6.85     0.99  Metadata Total
    
    Reviewed-by: Tino Reichardt <milky-zfs@mcmilk.de>
    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
    Closes #14746

As a reminder:

The size of a block is described by three different fields in the block pointer; psize, lsize, and asize.

lsize: Logical size. The size of the data without compression, raidz or gang overhead.
psize: physical size of the block on disk after compression
asize: allocated size, total size of all blocks allocated to hold this data including any gang headers or raid-Z parity information