RAIDZ3 12-wide: 4K files only ever read from 3 disks — does RAIDZ not rotate parity?

[tijanie]

I have a 12-disk RAIDZ3 pool. When I read a large number of small (~4K) files, only 3 of the 12 disks show any read activity — the other 9 sit at 0% utilization the whole time. I always assumed RAIDZ rotated parity across all the disks (like RAID5/6 does) to spread load evenly. But this looks like the data for these small files is permanently pinned to the same 3 disks.

[scineram]

Yes. The width should be relative prime to parity+1

[amotin]

RAIDZ does not rotate parity explicitly, relying on entropy from variability of block sizes after compression, metadata blocks, etc. If you look into vdev_raidz_map_alloc(), you'll see an artifact, when RAIDZ1 periodically swaps first two disks in each write to reduce the effect, but it was later considered pointless and does not apply to RAIDZ2/3. 12-disk RAIDZ3 is a very space- and performance-inefficient way to store 4KB files. I would not use it for blocks of less that ~256KB, if ever. IMHO if RAIDZ2 is not enough, you should better focus on some reliable reservation strategy, since "RAID is not a backup".

[tijanie]

Due to the skip sector appending a record to make it divisible by (nparity + 1), 4k record data will always land on the same 3 disks.

[amotin]

That's a good point. Another reason to not use this topology for 4KB files. For bigger and variable-sized blocks the padding will also try to align parity to the same disks, but it will have 3 choices (12 / (3 + 1)) used randomly.