How do you use zfs rewrite on volumes

[Gerporgl]

I am using zfs version 2.3.4, and I have the new zfs rewrite command, however it seems to only be useful for actual files, meaning filesystem type of storage. Since I am using zfs mostly on Proxmox VE, most of my data are under raw volumes.

Is there a way to use the zfs rewrite to rewrite an entire volume data?

Or will this be covered in version 2.4 with the new -P command? Or will this command still only work for filesystem?

[robn]

For now, zfs rewrite is only for filesystems. There's no particular technical reason it couldn't be extended to volumes, but I don't know if there are any plans to do so, and I haven't thought at all about if that even makes sense.

[Gerporgl]

I see, so even with the new -P command it would still not rewrite the entire pool?

Perhaps it does not makes sense. I just started to read about the rewrite command after I expanded my raidz pool to add another ssd and I thought I understood there may be some reason to do a rewrite after you expand, but perhaps there is no reason to do that.

Thanks for your inputs

[robn]

Yeah, these two do come up in conversation together, but they're not directly related.

When you expand a raidz vdev, new blocks written after the expansion can use the wider stripe size that spans all disks. Blocks from before the expansion are not rewritten, so they remain with their original stripe size. There's no problem for redundancy, but it can mean reduced efficiency over a vdev that was originally built with the full complement of disks.

When a block is overwritten, the old version is freed, and the new one is written down with the new stripe width (and other properties). This happens naturally as files are updated, but zfs rewrite gives a way to force blocks to be "overwritten" with the same data, so updating them for the new settings.

(Put another way: zfs rewrite does what you imagine might happen if you cp'd a file to itself, were that actually possible. But a bit smarter and safer, because OpenZFS internally understands what you want and can make sure various locking and atomicity guarantees are upheld.

You don't have to use rewrite, and it may not even be worth it; depends on your use case. For volumes it feels like it might be unnecessary, simply because they already have a relatively small block size, and also because volume blocks are frequently overwritten anyway. Like I say though, I haven't really thought much about it, or whether it's even worth building.

[Gerporgl]

Thanks for all the great info and explanation.

I just wasn't sure if expanding a raidz array was rebalancing all the data to all drives equally, I was assuming myself that the resilvering should be doing that, but I got confused after reading some other posts and comments on reddit and other places.