RAIDZ Expansion feature discussion

[justinclift]

This is carrying on from PR #15022, so we don't crowd the PR out with general discussion traffic. 😄

[yorickdowne]

So to summarize that: Afaik, a copy command will rewrite the data with the new width after expansion. I think I saw a script sloshing around that basically does that for an entire array, cleaning up after itself. Does it file by file for space reasons. Takes a minute, but, you know. Only have to do it once.

From my perspective that’s plenty good enough. Raid expansion is a niche feature, if much beloved by that niche. Adjusting width is then a niche within the niche. Handling that with a community script sounds like an entirely reasonable thing to do. Permissions and paths stay as is, filenames stay, last accessed and creation date changes and width changes.

To explain the niche comments: I am part of that niche. Home user but without the funds to start at full width. I did a poor man’s raid expansion eventually, from 5 to 8. So the niche are people who start at a smaller width than they know they’ll need, and will only run one vdev. By definition home users, and then only a fraction of those.

The niche within the niche are users where rewriting archival data makes an appreciable difference, or that care even though it doesn’t.

That need not be handled in the file system. Handling it with good UX outside seems great. Setups that use ZFS like TrueNAS and Proxmox could even integrate such a script into their UI.

[IvanVolosyuk]

@TurkeyMan
The 'assumed parity' is explained here by Matthew Ahrens after this #12225 (comment)

[Dulanic]

I'm part of that niche crowd, and I'm cool with it. If I decide to expand, I'm fine with keeping the old setup until it gets a rewrite. Storage is pretty cheap these days, which is why I'm looking to expand. That being said, I think my use case is decent. Right now, I've got two vdevs, but one of them has a bunch of older, 8TB drives. It just doesn't make sense to swap an 8TB with another 8TB when a 18TB is way cheaper per TB. As such, it just makes much more sense to expand my current vdev with 18TB drives and end up removing my old vdevof 8TB drives.

So, I much follow the path that you did.

With that said, when I remove my 2nd vdev, my understanding is zfs will shift my data over to the 1 remaining vdev. If this code/process exists, couldn't that process be called, albeit maybe modified, to balance it out again when expanding? Maybe not since its just moving the data from vdev to vdev vs adding a drive to expand which I guess would be a very different flow. This is where I may be a bit lost.

[IvanVolosyuk]

You can swap individually each 8TB with 18TB drives and once all of them are 18TB you can expand them to occupy entire space of the new drives.

[nonlinearsugar]

This script, with caveats (read the readme), helps automate rewriting your data by copying/pasting/deleting/moving at the filesystem level.

https://github.com/markusressel/zfs-inplace-rebalancing

[Dulanic]

FYI, Link is bad - goes to https://github.com/openzfs/zfs/discussions/url

[nonlinearsugar]

Fixed. Thanks.

[the8472]

The script doesn't use direct IO which means the data might end up sitting in caches and you're only hashing stuff from the page cache, not reading back what makes it to disk. I think this leaves a small window for data corruption, especially on systems without ECC memory.

[ahrens]

This is cool, and I think solves the problem well if you don't have snapshots.

If you wanted to preserve space sharing between snapshots and clones, you could use zfs send to copy entire "clone families" (all filesystems and snapshots related by cloning and snapshotting). The downside would be that you'd need enough free space for another copy of your largest "clone family", rather than just your largest file. The zfs send approach would also work with zvols.

[yorickdowne]

@Dulanic you can’t removed a raidz vdev from a pool. Mirror vdevs only (single disk vdev is special case of mirror) can be removed.

You may be able to create a new pool with your 18TB drives, copy data over, then expand that. This really depends on how much you’re using already though. This is why a lot of home users stick with one vdev, lest they are “stuck” with two to manage.

I have one 8x8 raidz2 which will likely last me forever. But in case it ever doesn’t, I intend to replace drives as they fail with larger-capacity ones. Once the last drive has been replaced, I’ll have more space.

[Dulanic]

Oh no! Lol I am still learning! That's a problem lol I have 80TB already! If I have 2 vdevs can I expand 1 if them? I wasn't sure if I needed to match the width.

[TurkeyMan]

My current take on this, the stripe-width expansion seems less of an urgent issue than the fact it only adds one drive at a time.
As advertised, at 50mb/s, it will take 4 days to expand my vdev. I can handle it taking 4 days I suppose, but the real problem is it can only expand one drive at a time. So, expanding 4-wide raidz2 to 8-wide, that'll take 2 or 3 weeks... and it's not clear how much the standard operating performance may be degraded during that time? Is there any additional risk from drive failure during that time?
It's a very long timeframe to expand a vdev.
It seems that this feature really needs to allow adding multiple drives at once...

[yorickdowne]

Unsurprisingly people have been making wider raidz2 vdevs for years, even without official expansion support. The “use sparse files for parity and offline them” trick has gotten good mileage :).

[ahrens]

FYI: Time to expand is limited by CPU usage. Improving it by 10x would be a relatively straightforward follow-on project. Also note that the 50MB/s that I measured was with 512-byte sector size. With 4K sector size it would be around 8x that (~400MB/s).

[justinclift]

Improving it by 10x would be a relatively straightforward follow-on project.

Is that something likely to be done in the near-ish future? 😄

[darkbasic]

Improving it by 10x would be a relatively straightforward follow-on project

I think that would be make raidz expansion much more viable.

[ahrens]

Is that something likely to be done in the near-ish future? 😄

Not by me, but PR's are welcome 😄

[GrayXu]

Is RAIDZ "contraction" support included in this PR? 🤔

[GrayXu]

Reasonable.
But since not all data in a zpool requires the same level of protection, I think the ultimate solution should allow different datasets to have different levels of parity protection. Like in the existing RAIDZ expansion, not all data has the same number of parities for protection.

[yorickdowne]

I’d really rather not see any such thing. ZFS is wondrously complex as is. Per-dataset parity adds another level of complexity for no gain that I can see.

In this expansion PR, all data have the same parity in the expanded vdev. It’s the stripe width that changes, not the parity.

If I recall right there has been discussion of a follow-on PR that’d expand (but not contract) parity, such as from raidz1 to raidz2 or raidz3. That’d be separate work, and whether it will ever be tackled is a good question.

The existing mechanisms for changing pool layouts - make a new pool, and/or add vdevs - work for the use cases I can see. Raidz expansion is a lovely hobbyist use case and I am stoked it’s getting there; raidz parity expansion would be another hobbyist use case but even more niche.

[amotin]

The way expansion is implemented now the contraction is simply impossible. Even in unlikely case that end of the vdev is absolutely empty to be dropped, contraction would not produce sufficient level of redundancy to sustain disk failures.

[GrayXu]

@yorickdowne @amotin thanks for reply!

1. Regarding the issue of data reliability. The number of data dev and parity dev are both related to reliability, and expanding stripe width will also reduce system reliability. Similar topics are widely researched with different coding parameters in erasure coded systems.
2. Regarding different datasets having different levels of protection. Obviously, we can achieve different levels of protection through manual partitioning to different zpools, but this cannot achieve the effect of sharing the same storage space. I think it is reasonable to have different types of files requiring different levels of protection, such as both 3-way replicas and tape replicas in data centers. So I am particularly curious about the engineering challenges of this on ZFS.

[justinclift]

@GrayXu While some of that sounds like a potentially interesting discussion topic, it's probably best to create a new discussion for it. It's not very on-topic for this one. 😄

[yorickdowne]

To @farkeytron: Discussion is here, not in the PR. IxSystems is paying for this work, so it’ll definitely come into TrueNAS. SCALE first likely, since that is where canary features land, then Core in due time.

[kmoore134]

Lol, very appropriate meme :)

This feature will involve more middleware and UI changes to support properly, which means it will be targeted at SCALE only. CORE is mostly feature frozen, so we aren't making big sweeping changes to features / backend / front-end. SCALE is where active development resides.

[MihaMarkic]

"next major update to our ZFS base" What timeframe is this? Next major build after Cobia? Sorry if I've missed this info.

[kmoore134]

24.10 (Electric Eel - October release) is the next major TrueNAS SCALE release that will include ZFS 2.3.

[CBattershillDev]

Reading this makes me want to upgrade my core server to scale but migrating things like nextcloud configurations and data makes me concerned. Is it a certainty that this feature wouldn't make its way to core? (even if it was just for cli access)

[kmoore134]

That feature won't make it into 13.3, so odds are low it will land in CORE in the next 12-24 months. If you need a hand with migrating nextcloud we've had plenty of other folks do that. Might be worth asking questions here:

https://forums.truenas.com/

[KungFuJesus]

we have this issue with ZFS encryption support after Datto stopped contributing to it.

@bghira Is there more info on that around?

Asking because I'm using ZFS encryption on some systems, and hadn't heard mention of potential upcoming problems with it. Would like to know before problems potentially bite. 😄

Hah, I take you don't follow the issue tracking on github too carefully, then. The amount of issues I see with the encryption support (and not just performance problems, but like, swallowed my data or corrupted it on a replication ones) has made me pretty hesitant to recommend ZFS's native encryption support for anyone.

[justinclift]

Hah, I take you don't follow the issue tracking on github too carefully, then.

Yeah, not at all. I think this discussion and it's matching PR are the only things I'm subscribed to in the repo.

swallowed my data or corrupted it on a replication ones

Damn. Sounds like we'll need to investigate that, and potentially find an alternative to ZFS then. Or at least an alternative to native ZFS encryption. 😦

Can't be having unreliable storage backend for production systems. 🤦

[darkbasic]

I use encryption in production: it's not worse than many other features like zvol (#11407).
zfs is not perfect, but what's the alternative? dm-crypt + grub2 patched for argon2? I find it extremely irritating that without out of tree patches your system is as secure as one without encryption. If I have to sign my own bootloader (because I have to build grub from source) I'd rather sign a zfs module instead.

[justinclift]

@don-brady @ahrens Thank you SO much for getting the ZFS expansion PR across the line.

It'll likely be a key capability of ZFS from here on (in the next release onwards of course). 😄

[Sparklingx]

is the expansion option for all setups including stripe?

[mumbleskates]

it's for raidz vdevs specifically, which have 1-3 devices worth of parity.

[KaeTuuN]

This Feature is not for switching from stripe to raidz1-3!

[IvanVolosyuk]

Quick question: if I update to zfs master and enable corresponding feature flag for raidz expansion, will it be active or enabled after expansion completes? Thus, can I downgrade to stable branch after that or there will be some incompatibilities in on disk layout and I'm limited to master branch after that until the next stable release.

[amotin]

On-disk format after expansion is different from one supported before this feature. So I bet the feature stays active forever.

[ahrens]

That's correct. After expansion the feature will be "active", and you can't downgrade the software to an earlier version that doesn't support RAIDZ Expansion.

[emaste]

A couple of days ago @mmatuska updated OpenZFS in FreeBSD main, including RAIDZ Expansion. You can try this out with the next set of 15-CURRENT snapshot images.

freebsd/freebsd-src@e716630

[badmannen]

I tried to read through the DOCS here but still not sure if I can extend my 2-disk VDEV ( only option to create was a mirror ) Can this be expanded with a 3rd disk as RAIDZ-1 ?

[yorickdowne]

No. A mirror vdev is not raidz-1. But this is what you can do:

Break the mirror, remove one drive. Take that drive, a second fresh one, and a sparse file, and make your raidz1 in a new pool, not the same pool! Offline the sparse file.

You now have two pools with one vdev each, and no redundancy. Hope you have a good backup.

zsend/zrecv from the old pool to the new one

verify everything is there, twice. Because the next step means you can’t go back.

destroy the old pool, replace the offlined sparse file in the raidz1 with the now free drive. Let the raidz1 resilver.

This would also work with a raidz2 and two new drives. Which is a bit more future proof. Raidz1 with larger drives has a tendency to break during resilver.

[badmannen]

With my N00b level of understanding you did a great job of explaining. With that said I think I lean towards Backup-copy My 2 disk mirrored Vdev, then create a brand new with my now 3 disks and then copy back the data into the new vdev? ( more failsafe and faster ? or am I missing something , got 14 TB disks)

[yorickdowne]

With 14TB disks raidz-1 is risky when rebuilding. I recommend you do a 4-wide raidz2, which you can then expand later if desired.

[badmannen]

Sounds like a good solution ... just need to squeeze my wallet upsidedown and hope some monetas will fall out from it =) ( but yes If I don´t I will probably regret it later on )

[yorickdowne]

“Buy nice or buy twice” :)

[DurvalMenezes]

This is arriving at a really good moment for me, I was about to expand a 4-disk raidz2 into a 6-disk by the old method (create an entirely new pool then zfs send | recv from the old one into it), so if this is implemented and working, it's going to save me a ton of time and effort.

So I must ask:

1. IIUC what I read here and at the PR, it's implemented on zfs-master, right?
2. Is there an estimate for when this will be available in a main release?
3. Has anyone used it already? How did it go?

TIA!

[DurvalMenezes]

Thanks @yorickdowne and @darkbasic for your responses.

And I just read a little more about the limitations, and it seems the data-to-parity ratio isn't recalculated either, so a ton of space would end up being lost if the expansion is done using this new functionality.

Seriously, we've been waiting since forever for... this? :-( don't want to sound ungrateful but it's really disappointing to see that ZFS remains ITR so far behind what other folks (eg Linux md) have been doing for literally decades.

Anyway, back to the ancient zfs-send-recv-into-new-pool I go...

EDIT: the situation is much less worse than I was led to believe, see my next comment below,

[darkbasic]

it seems the data-to-parity ratio isn't recalculated either

That is just until you rewrite the existing data. If your data is going to change over time (or if you don't care about snapshots and simply rewrite it) you will regain the lost space.

[ahrens]

I might very well be wrong but I think you won't be able to expand existing arrays, only new ones created after the feature has been enabled.

That is incorrect. You can expand existing RAIDZ groups.

[darkbasic]

That is incorrect. You can expand existing RAIDZ groups.

That is awesome, I've never been so happy to be proven wrong :)

[DurvalMenezes]

Thanks @ahrens and @darkbasic for the additional corrections/clarifications. So the situation is much better than I was led to believe.

The major (only?) conses seem to be the (a) inability of the upgraded array to be handled by previous versions of the software not supporting this feature, and (b) desirability of rewriting all data in order to "rebalance" already-existing data (which I understand means both to redistribute its data on the new array including the added vdevs, and recalculate its parity blocks for the new scheme).

(a) is not a major problem (just test the new version well enough to be damn sure no version downgrading will be necessary) and (b) is just an inconvenience for my use case (I presume that, once I have the necessary disk space, I can just shuttle the data inside it from the previous datasets to new ones to force the rewriting with zfs send|recv, and then destroying the pre-rebalance datasets and renaming the new ones to the old names).

I will now mark this on my original comment above.

[jakesteele]

How does expand address fragmentation, if my cap is under 50% does the end result remove most/all fragmentation? What if my capacity is over 50%, does the end result in more fragmentation then before?

[ahrens]

The newly-available space will be one big free chunk, but expansion doesn't change fragmentation of the existing data.

[jakesteele]

Gotcha, I figured expansion when re-writing between disks would fix fragmentation if the free space was greater than 50%.

[GottemHams]

I'm considering upgrading the disks in my system with something faster, and add an extra disk in the process. So let's say I first add the extra disk and let ZFS expand the pool, then swap out the pre-existing disks one by one (or perhaps connect the faster one alongside) and let it resilver each time. Would the resulting pool have the updated data-parity ratio? I would assume so since eventually all disks would have newly written data, but because it happened in response to a resilver it may not take the earlier expansion (fully) into account? As in: disk X is pretty much copied bit-for-bit to replacement disk X, resulting in the same original ratio (only that disk has to be rebuilt so there's no (re)balancing of data-parity). I don't think there's enough room to connect all the new disks alongside the original ones and just send it over. :DD

[Cosine5000]

You realize this functionality is still a year or two away, yes?

[GottemHams]

Ever heard of planning? If the scenario I described wouldn't work then it's probably better to find a way to make room for a new pool and send over the data after all, which I could do right this moment. Otherwise, having to wait for 1-2 years with the current setup is perfectly acceptable.

[Quadrubo]

I'm planning to use ZFS for my homeserver with proxmox. Would it be possible to setup a 2 Drive RAIDZ1 right now and add a third drive later using this method when it releases?

[justinclift]

Yeah, that's the kind of thing this is for. 😄

[Quadrubo]

Awesome thank you!
Wasn't even sure if RAIDZ1 with just 2 drives is a valid configuration.

[justinclift]

Oh, that's a good point. I think the smallest # of drives I've tried with a RAIDZ1 is actually 3. I'm not actually sure about 2. Err... oops? 🤦

[IvanVolosyuk]

2 drive raidz is just a mirror with crippled performance. You better of creating real mirror now and one you have all 3 drive - split the mirror and create degraded raidz from 2 of 3 drives and zfs send all the data from the 3rd. After that you can attach 3rd to make it redundant again. You will sacrifice redundancy during migration though.

…

On Thu, Feb 29, 2024, 02:55 Quadrubo ***@***.***> wrote: Awesome thank you! Wasn't even sure if RAIDZ1 with just 2 drives is a valid configuration. — Reply to this email directly, view it on GitHub <#15232 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HJZEA2NGDSHANUI3LDYV5HP7AVCNFSM6AAAAAA4HOSFKSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DMMRQGM3DM> . You are receiving this because you commented.Message ID: ***@***.***>