Handling a dynamic (adding/removing/replacing drives) filesystem with snapshots

[Watkurem]

I've been successfully using bees on my backup server for a long time now. Recently, however, I've had the data amount grow far enough to make hash table size not optimal, attempted to increase its size, and now it seems like I can't use bees anymore, as it is stuck deep in snapshots and seems keen to grow my metadata to uncomfortable amounts.

I turned the service off for now. I don't think any of this is bees's fault, as I brought it all upon myself. But I am wondering whether there's a way to recover (and whether I even should, as bees might simply be a bad fit for my usage scenario).

Background

Some years ago, I built myself a backup server with a single 8 TB drive (I am only mentioning the storage drives; OS lives elsewhere and is not relevant, as it's not even on btrfs). I figured that, if I rsync my other machines to subvolumes on this server, and then create readonly snapshots of those volumes, I'm going to make great use of btrfs's CoW features. To maximize CoW, I used the --inplace --no-whole-file -M--no-whole-file rsync flags, to make sure that, if a file is only partially changed, it will be only partially overwritten.

This seemed to work well enough. I wanted to optimize further, so I enabled compression (compress-force=zstd:12) and set up bees. (I also did other things, like using bcache with redundancy and a custom kernel with kakra/linux#36, but they don't seem relevant, so I'm not mentioning them here.) When setting up bees, I did some research and accounted for a couple of things:

1. While bees can be used with however many snapshots one wants, the snapshots should be created after dedup. Because of this, I made sure to first do the backup rsync, then let bees run, and only then snapshot once bees is done.
2. The optimal hash table size can be calculated as unique_data_bytes / (128 * 1024) * 16 (amount of 128K unique hashed extents times 16 bytes per entry). I figured that 8 TiB of unique data is a reasonable expectation with an 8 TB filesystem (that is expected to grow later on), so I went with a 1 GiB hash table.
3. It's pointless to dedup (and, indeed, compress) some files (e.g. VM storage) to avoid insane fragmentation, so I made sure to mark such files nodatacow.

It all went fine and performed about as I expected. Then I grew the filesystem, adding two more 20 TB drives (for a total of 48 TB raw storage and 24 TB usable space in a raid1 configuration) and ran a full balance as is recommended. Then my amount of data grew steadily as I made more backups, at some point reaching current 10.5 TiB, and I wondered whether the hash table size is future proof.

Failure of cognitive function

Despite 10.5 TiB being still perfectly fine and within tolerance for a 1 GiB hash table, I, being the smart and insightful professional that I am, considered the possibility that it will be too costly to grow the hash table once data amount reaches, say, 40 TiB. So I decided to grow the hash table to 4 GiB preemptively, as I have the RAM to spare. I looked up how to do it, found this issue #70 (and some other ones, which said basically the same things I believe), and learned that I have to delete beeshash.dat and beescrawl.dat and change the db size in the config.

Of course, I completely failed to understand or consider the implications. Deleting those files basically reset bees and made it crawl and dedup my whole storage from scratch — that is, 100+ snapshots of 3-4 TiB each. I have also completely missed (or forgot, since I researched this so long ago) that bees will create its own metadata for every snapshot as it dedups.

I got alarmed after my metadata tripled in size, did some better research, stopped bees, and here we are.

What next?

Accepting the damage (extra metadata) and moving on without bees is the most obvious default choice. My usage scenario probably makes bees less useful than it could be, anyway, as the in-place writes and incremental backups deduplicate things quite a lot as is.

But what if I wanted to keep using bees? I thought a bit and came up with a "replay step-by-step" plan:

1. Making use of LVM features, shrink the existing btrfs filesystem and create a new one beside it on the same set of drives, with same settings.
2. Replicate the backup subvolume structure (1 subvol per machine, separate nodatacow directories for troublesome files, etc.).
3. For each backed-up machine, rsync the oldest history snapshot into the new filesystem. This should fully defragment the data and minimize metadata.
4. Run bees on the new filesystem. Wait for it to finish dedup.
5. Create snapshots of the deduped subvolumes, replicating the oldest snapshots within the new filesystem.
6. Delete these oldest snapshots from the old filesystem.
7. GOTO 3 until there are no snapshots left in the old filesystem, balancing and shrinking it on the way, as needed, to grow the new one.
8. Delete the old filesystem and replace it with the new one, fully deduped with new proper hash table size.

So, basically, I would replicate the steps I took to create each backup in the first place, one by one. This is obviously very involved and will take a very long time, but it seems like the best option I've got (aside from deleting the snapshots and starting anew from the current state).

This can be made potentially faster and more efficient if I segregate the backups. As I don't really need most of this data in versioned history, I could shrink the snapshots quite a lot by storing the non-versioned data separately, in subvolumes without snapshots. Finally, I could even keep the non-versioned data on a separate btrfs filesystem, and run bees only there. Which brings me to...

What's the right way to handle a dynamic (adding/removing/replacing drives) filesystem, with snapshots, with bees?

Going back to #70 and stuff mentioned there (once again, there were other issues I looked through, but the info was more or less the same), it seems like:

• The only current way to resize the hash table is to nuke it. Along with it, one must nuke the crawl state, because otherwise dedup won't be efficient, as it won't be aware of any older data (at least that's how I understand it).
  So, the only way to resize the hash table is to basically start bees from scratch, which should not be done if you have a lot (or even a few, really) of huge snapshots.
• Full balance (like one must do when adding/removing drives or replacing faulty ones) invalidates the hash table. Once again, one must drop the crawl state along with it to avoid the same issue (dedup being inefficient because it can't find the older data to reference).
  So, a full balance for any reason also basically forces one to start bees from scratch.

If the hash table sizing issue can be circumvented (predicting the expected data size and picking a hash table size accordingly is a reasonable expectation, especially since there's no hard requirement to keep the optimal 128k dedup extent size), the balance issue seems just completely unavoidable with a multi-drive storage system (even if one never adds or removes drives, a drive will fail at some point, requiring a replacement and a full balance).

So does that mean bees shouldn't be used on multi-drive storage with many (and/or huge) snapshots? As a balance will be required at some point, which will invalidate the hash table, which will either cripple bees or require restarting from scratch (taking ages and exploding metadata due to snapshots).

Or is there a right way that I'm missing? Maybe this info is just stale and balancing is somehow handled without hash invalidation by now? Or maybe there's some way one can run a one-off "hash table fill" that will read through all extents (rather than following snapshots and transactions) and recover the hash table to a usable state in reasonable time?

[Zygo]

To maximize CoW, I used the --inplace --no-whole-file -M--no-whole-file rsync flags, to make sure that, if a file is only partially changed, it will be only partially overwritten.

This can create a large number of unreachable blocks as snapshots are removed, and reduces rsync efficiency over slow networks. I run rsync without --inplace and let bees determine how to reuse existing extents without creating large amounts of unreachable data.

1. While bees can be used with however many snapshots one wants, the snapshots should be created after dedup. Because of this, I made sure to first do the backup rsync, then let bees run, and only then snapshot once bees is done.

There's no need to pause bees for this. bees can dedupe rsync's temporary files while it's still writing them. There's a relatively small metadata impact if the snapshot occurs right at the end of the rsync--as long as bees completes dedupe before the following snapshot is created, there will be at most two references per deduped extent--usually less, unless your files are predominantly small.

The situation to avoid is having many snapshots build up before dedupe has completed on any of them, as the metadata size is multiplied by the number of referencing snapshots when dedupe occurs after the snapshots were created. A multiplier of 1-2 (snapshot created immediately after rsync, while dedupe is still running) is negligible compared to the total filesystem size, but a multiplier of 100 or 1000 can be a problem.

3. It's pointless to dedup (and, indeed, compress) some files (e.g. VM storage) to avoid insane fragmentation, so I made sure to mark such files nodatacow.

v0.11 reduces the fragmentation considerably compared to v0.10, although there's still a "long tail" of small fragments in the compressed size tier. I haven't been able to find a one-size-fits-all approach to small fragments--in many (most?) data sets, they are 1% of the total size of the filesystem, but in some special cases (like Windows VM images), they're almost 10% of the filesystem and 30% of duplicates.

Multiple snapshots of VM images over time usually dedupe very well--usually producing data space gains fair in excess of the metadata cost.

Marking files nodatacow prevents btrfs from protecting the files with csums, so I never use nodatacow on a backup server.

We can make the fragmentation tolerance more configurable in bees. Right now it's an arbitrary constant--100 fragments or less is OK, 101 or more is not--but there are some workloads where it makes sense to set the limit much lower, like 10 new fragments, or 1, or even 0 to behave like duperemove.

It all went fine and performed about as I expected. Then I grew the filesystem, adding two more 20 TB drives (for a total of 48 TB raw storage and 24 TB usable space in a raid1 configuration) and ran a full balance as is recommended. Then my amount of data grew steadily as I made more backups, at some point reaching current 10.5 TiB, and I wondered whether the hash table size is future proof.

If you are incrementally filling the filesystem with new data each day, it's probably an OK size for rotating snapshots (where old snapshots are removed as the filesystem filled up). For archiving snapshots (where old snapshots are never removed, or it's more important to dedupe the oldest data against newest data), it may be too small.

The hash table is a sliding window over the last data that was scanned. Duplicate data tends to occur in clusters over time, so there is usually less value in recalling older data as it ages. If you have an old piece of data that keeps reappearing, bees will refresh the hash table entries each time it is encountered, keeping it in the window while expiring old unique data that doesn't reappear.

The main driver for increasing the hash table size are when the distance between duplicate extents gets too large to fit in the table, or you have a very high dupe rate (75% or more) on a subset of the data that is larger than the hash table. e.g. if you have 1 GiB of hash table and you are adding more than 256GB to the filesystem per update cycle, the hash table window would be flooded by new data and lose an unreasonable number of dedupe opportunities.

I decided to grow the hash table to 4 GiB preemptively, as I have the RAM to spare. I looked up how to do it, found this issue #70 (and some other ones, which said basically the same things I believe), and learned that I have to delete beeshash.dat and beescrawl.dat and change the db size in the config.

Currently that's the only way to do a hash table resize. Resizable hash tables are possible without losing all the state, but the old table entries have to be inserted into the new table in the right order--it's not as simple as truncating the hash table file to a larger size. It's becoming the most frequently requested feature.

Of course, I completely failed to understand or consider the implications. Deleting those files basically reset bees and made it crawl and dedup my whole storage from scratch — that is, 100+ snapshots of 3-4 TiB each. I have also completely missed (or forgot, since I researched this so long ago) that bees will create its own metadata for every snapshot as it dedups.

The metadata is created by btrfs when a shared subvol page in a snapshot is converted into reflinks on unshared subvol pages. This is not specific to bees--merely touching a file in a snapshot (or even reading the files in a snapshot, if the filesystem is mounted with -o relatime and the snapshots are read-write) has a similar effect. bees makes the normal conversion of snapshot pages to reflinks run much faster, but it cannot be stopped entirely without stopping almost all writes to the filesystem.

But what if I wanted to keep using bees? I thought a bit and came up with a "replay step-by-step" plan:

[....copy the entire filesystem to another filesystem by brute force...]

I've done that to work around old btrfs metadata bug that damaged a filesystem. I did the steps in a different order:

1. Start bees on the receiving filesystem and keep it running for incremental dedupe. This was important in my case since the data would be too large to fit on the target filesystem without dedupe.
2. Copy each series of snapshots (i.e. the snapshots for machine A) in reverse order. Copy the newest first, then snapshot to make a subvol for a previous version, then rsync to update the previous version, repeat. This is based on the assumption that files grow over time, so older files will be shorter, and can be created by truncating newer files.
3. Verify each transferred snapshot by comparing sha256sums.
4. Remove old snapshot.
5. Repeat until original filesystem is empty.

So, basically, I would replicate the steps I took to create each backup in the first place, one by one. This is obviously very involved and will take a very long time, but it seems like the best option I've got (aside from deleting the snapshots and starting anew from the current state).

The problem with this plan is that the metadata size will inevitably grow again. A filesystem filled by plain rsync with snapshots and no dedupe is the smallest possible metadata case, but metadata will continue to grow as long as at least one subvol in a snapshot series is updated. So with this plan you might do a lot of IO, but achieve only a temporary reduction in metadata size.

You are adding many TiB more space to the filesystem, so there is room for some metadata expansion--though we probably want to avoid having it grow by a factor of 100 if we can, which could happen if there are many dedupes missed by previous scans.

So does that mean bees shouldn't be used on multi-drive storage with many (and/or huge) snapshots? As a balance will be required at some point, which will invalidate the hash table, which will either cripple bees or require restarting from scratch (taking ages and exploding metadata due to snapshots).

No, quite the opposite. I run bees on 100+ TiB filesystems with 10+ drives, and routinely swap out the drives as they age and rebalance as the data grows.

bees v0.10 and earlier were not fast enough to keep up with a balance, so a full balance would invalidate the majority of the hash table before bees could update the hash table with the new locations of data. v0.10 and earlier would fall behind with no possibility of catching up with the balance, so bees's IO was wasted, and slowed the balance down.

bees v0.11 scans faster than btrfs balance can relocate data, so you can simply leave bees running during balance, file resize shrink, device remove or replace, or automatic bg reclaim. bees will keep the hash table up to date in near real time, fitting bees's IO into the idle periods of balance's IO.

If you stop bees before the balance, and you don't start it again until after balancing all of the data, then you've recreated the v0.10 "too slow to keep up with balance" situation. In that case, it would be better to wipe and rebuild the hash table, as there's nothing usable left in the hash table, and all the data in the filesystem will be scanned as new data whether you restart or not.

Or maybe there's some way one can run a one-off "hash table fill" that will read through all extents (rather than following snapshots and transactions) and recover the hash table to a usable state in reasonable time?

That seems like a good idea for handling the cases where bees is started on a filesystem with many snapshots that haven't yet been converted to reflinks. I'm presently working on loop controls (to handle the "run one scan pass and stop" and "run until done then stop" use cases), and a "only insert hashes during the first scan" mode would fit into that feature set nicely.

[kakra]

Here may be some interesting reads partially covering similar issues:

• What amount of work is redone after reboot? #320: For some reason I was able to help bees progress by selectively deduping snapshots with other tools, a combination of file-based dedupe (fdupes) and extent-based dedupe (duperemove) may give bees a better starting point, replacing older more complex generations with simpler fresh generations pre-deduped
• HDD performance decreasing a lot #298: forcing compression will considerably slow down bees and most likely increase the hash table size, you shouldn't try squeeze every single 128k (or lower) dedupe opportunity out of bees

So here's my take:

0. I'm pretty sure some of these steps may break btrfs-send/receive if you're using that. But it looks like you're using rsync.
1. Don't force compression, leave the default heuristics. Maybe even do not use very high compression levels to reduce the chance of compressing files which compress poorly anyways. You'll gain more from deduplication. A lot of tiny extents (due to compression) will increase the meta data overhead and counter the effects of compression.
2. fdupes is a full-file dupe finder. After turning off forced compression, create a rw-snapshot of your latest data, then defragment it selectively (to avoid generating too much new data), then selectively use fdupes against the same folders in other snapshots. This should recombine files from the snapshots into files with fewer extents. You can then remove the rw snapshot later. Hint: Avoid using fdupes against files which are changing, it uses an unsafe btrfs clone call although it tries hard to protect against cloning files which have been changed since scanning.
3. duperemove is an extent-based dupe finder. It will use a lot of RAM. Try using it selectively against contents which bees struggles with. It should catch the remaining extents which fdupes didn't find. Always use duperemove from the same base directory so the relative path to your data stays the same. It will accumulate files paths in the hash database over time, up to the point where it becomes incredibly slow and memory consuming. Start over with a clean hash file then.
4. Finally let bees run for some time again, see where it struggles and take note of some of the directories which show up often and for a long time in its progress file. Then repeat steps 2 and 3 for that data.
5. Eventually bees will suddenly start making much more progress in less time. This will only work if the above steps reduced the amounts of small extents (by turning forced compression off, defragmenting, and recombining identical files).

Hint: Do not use too big data sets with these steps as bees might stop progressing while you're introducing high file system loads. Otherwise, this may render most of your hashes in the hash table useless because bees has to go through too much new data when it continues.

@Zygo suggested that there's no point in running these steps because it only defers pending work for bees to later generations. And that's probably true for the most part. But still it helped me, probably because I managed to recombine many small extents back into bigger ones. That's where bees actually shines: It can scan big extents very fast and rewrite extents into different extents with higher probability of matches. But it will crawl very slowly with a lot of small extents.

I've lost my crawler state multiple times for different reasons, and these steps always helped speeding up bees progress to keep pace with snapshot creation again. So your situation may be similar. You're using a lot more snapshots, I'm only using 20-30, with time-based retention. So the process will probably be a lot slower for you, multiple weeks at least.

Since you're using my kernel patches, I hope you're using dedicated SSD/NVMe for meta data, which will speed up your use case a lot. If you can effort it, consider migrating meta data to dedicated drives at least during the process. bcache alone isn't very good at accelerating btrfs meta data. And bcache will be mostly useless anyways during this process because you will be handling lots of data which is read only once. So maybe set bcache to none to save some drive wear. Or detach the bcache (ensure there's no pending writeback, turn off writeback caching first), wipe the cache devices, make that the dedicated meta data disks, and later reintroduce the bcache again.

Turning forced compression off should also bring meta data usage down after the process is done. But you won't get back to the original size because, as you noticed, bees recreates a lot of meta data due to rewriting extents.

OTOH, since you added two more disks, you should still have enough space to use the process described by @Zygo: copy latest snapshot to the new drive, take a snapshat, replicate with rsync, repeat in reverse order. Your snapshots are probably in a very organized structure, so it should be easy to automate via a script, including the verify script, and waiting some time for bees to catch up with each new synced snapshot (which seems to be important to not bring it back into the slow state and lagging behind).

If you create a new btrfs, you can create a bcache backing store without cache attached, so you can later easily attach your cache again when the process is done.

@Zygo:

and a "only insert hashes during the first scan" mode would fit into that feature set nicely.

That sounds very useful.

[kakra]

@Zygo On another note: Couldn't that "rsync to new snapshot" be done on the same drive: first turn off forced compression to reduce meta data overhead and increase bees pace, then rsync oldest snapshot to an empty subvol in a new folder structure, make it read-only, delete the source snapshot, create "older" target snapshot (instead of empty subvol), then repeat until done? This should clean up meta data, and bees will progress faster and faster over time. I'd introduce some sleep time between each cycle to ensure that bees can keep up. As far as I understood, bees will scan new data as it appears before going back to older data, so it should be fine for the hash table. To be sure, one could set the crawl state to the current generation number first, so it will only scan from the current file system state and ignore the old stuff.

This way, an initial balance would probably not even be needed, as the copied data naturally balances over the new and old drives as the loop progresses.

[Zygo]

compress-force=zstd:12

Ouch, I missed that.

Definitely don't use compress-force if you're concerned about metadata size. It will hurt performance on reads and dedupe of uncompressed data. Use the compression attributes instead (chattr -R +c on the filesystem, excluding files that already have the conflicting +C attr, and use -o compress=zstd:12 for mounting to set the compress level) so that incompressible data can be written in large extents while using heuristics to avoid short extents. Or patch the kernel to increase the compression batch size (see kdave/btrfs-progs#960 (comment) for details) so that compress-force doesn't chop up the data into tiny pieces.

[Zygo]

Couldn't that "rsync to new snapshot" be done on the same drive: first turn off forced compression to reduce meta data overhead and increase bees pace,

To be clear, you'd still have compress=zstd:12 in the mount options, as there's no other way to set the compress level (except defrag on new kernels). Only -force needs to be removed.

As far as I understood, bees will scan new data as it appears before going back to older data, so it should be fine for the hash table.

Sort of, but there's a better way to do it.

bees will scan longer extents before shorter ones, so it is important to not use compress-force here. compress-force will prevent any long extents from being created, so bees won't be able to take advantage of longer the extent sizes. bees will be more likely to carve up a 512K extent into smaller pieces, since it only needs to find 256K of duplicate blocks, than it is to carve up a 128M extent, which requires 64M of duplicates before bees will touch it.

A better way to do it is....

To be sure, one could set the crawl state to the current generation number first, so it will only scan from the current file system state and ignore the old stuff.

...that. Wipe the hash table so bees doesn't know about any of the old, fragmented extents, and set min_transid to max_transid on every line in beescrawl.dat. bees will then start up and go idle because there's no new data. Then each rsync that occurs after that will be considered "new" and inserted into the hash table, and only the new extents will be considered for dedupe. If all of the existing data is copied with rsync and without compress-force, then the new copies will have nice large extent sizes, so bees will use them as dedupe sources and do fewer nuisance dedupes. Delete the old copies as you go, and their metadata will be removed with them.

You need a lot of free space to do this without the copies ending up fragmented as well, but you have that: ~16 TiB free space, if I'm reading correctly.

[Watkurem]

Thank you very much for such quick and detailed responses!

@Zygo

This can create a large number of unreachable blocks as snapshots are removed, and reduces rsync efficiency over slow networks. I run rsync without --inplace and let bees determine how to reuse existing extents without creating large amounts of unreachable data.

Huh. I came up with this approach before I learned about bees, so I just "put bees on top of it". You're talking about --inplace specifically, right? How does it hurt rsync over slow networks?

There's no need to pause bees for this.

My idea here was to reduce hard drive random access (deduping one file while writing another one intuitively sounds like a recipe for wrecking throughput).
Now that I think about it, though, this is probably handled by OS, especially with truckloads of RAM available (as I happen to have).

v0.11 reduces the fragmentation considerably compared to v0.10, although there's still a "long tail" of small fragments in the compressed size tier. I haven't been able to find a one-size-fits-all approach to small fragments--in many (most?) data sets, they are 1% of the total size of the filesystem, but in some special cases (like Windows VM images), they're almost 10% of the filesystem and 30% of duplicates.

Multiple snapshots of VM images over time usually dedupe very well--usually producing data space gains fair in excess of the metadata cost.

Marking files nodatacow prevents btrfs from protecting the files with csums, so I never use nodatacow on a backup server.

We can make the fragmentation tolerance more configurable in bees. Right now it's an arbitrary constant--100 fragments or less is OK, 101 or more is not--but there are some workloads where it makes sense to set the limit much lower, like 10 new fragments, or 1, or even 0 to behave like duperemove.

Some of this goes over my head, as I don't have all knowledge and context here. But I can say that I started doing this nodatacow thing after my windows VM images got split into literally millions of fragments at some point, and attempting to read them gave 1-2 MiB/s speeds. That, in turn, resulted in backups taking several extra hours just to transfer a tiny portion like that.
Removing the images and making fresh copies with nodatacow resulted in proper 150-200 MiB/s speeds that the hard drive is capable of.

As there's no point in a backup that I cannot read in reasonable time frame, I opted for not keeping history (and checksums) of the VM images in favour of overall speed.

This fragmentation could've been because I also backup windows machines to the same server, so bees probably deduped images with those separate files (rather than just deduping them with each other).

And... My server is still on v0.10, huh. Time to upgrade.

If you are incrementally filling the filesystem with new data each day, it's probably an OK size for rotating snapshots (where old snapshots are removed as the filesystem filled up). For archiving snapshots (where old snapshots are never removed, or it's more important to dedupe the oldest data against newest data), it may be too small.

I do weekly backups and archive until I run out of space. Then the plan is to either add more drives, or remove old snapshots. So far I've been adding drives 😁

A single snapshot contains several TiB of data, but the change each week is relatively small (although I might add several hundred GiB of new data at a time, this data tends to be video and other compressed non-dedupable stuff).

The hash table is a sliding window over the last data that was scanned. Duplicate data tends to occur in clusters over time, so there is usually less value in recalling older data as it ages. If you have an old piece of data that keeps reappearing, bees will refresh the hash table entries each time it is encountered, keeping it in the window while expiring old unique data that doesn't reappear.

The main driver for increasing the hash table size are when the distance between duplicate extents gets too large to fit in the table, or you have a very high dupe rate (75% or more) on a subset of the data that is larger than the hash table. e.g. if you have 1 GiB of hash table and you are adding more than 256GB to the filesystem per update cycle, the hash table window would be flooded by new data and lose an unreasonable number of dedupe opportunities.

I'm now realizing that I didn't understand how the hash table works at all. From table in https://github.com/Zygo/bees/blob/master/docs/config.md#hash-table-sizing, I've somehow been assuming it's a rough mapping of the whole file system (and as the file system grows, the mapping gets less precise). In hindsight, that... That just wasn't smart of me; how could bees do block level dedupe if it didn't keep a hash per block?

If a hash is per block, my 1 GiB hash table was able to cover about 256 GiB of data (4k blocks), you're quite correct! The new 4GiB table will cover 1 TiB, which should be well within my expectations for new data introduced per backup.

Carefully reading (rather than just glancing at the sizing table with the assumption that I've figured it all out before...) through the config.md, I can see that there's an extent scan mode, which I am seemingly not using. Because it is probably not part of v0.10. I imagine that added to my problems... Quite a bit.

The metadata is created by btrfs when a shared subvol page in a snapshot is converted into reflinks on unshared subvol pages. This is not specific to bees--merely touching a file in a snapshot (or even reading the files in a snapshot, if the filesystem is mounted with -o relatime and the snapshots are read-write) has a similar effect. bees makes the normal conversion of snapshot pages to reflinks run much faster, but it cannot be stopped entirely without stopping almost all writes to the filesystem.

My snapshots are readonly and are not changing. However, after dropping the hash table, the metadata started growing rapidly while bees was chugging through the snapshots one after another. I started with just 60 GiB of metadata for the whole storage (this was already "deduplicated" as bees was running while backups were coming in), and after 3 weeks of non-stop hdd activity at sustained 400 MiB/s of crawling ended up with 180 GiB with no end in sight.

I figured this was because of bees deduplicating each snapshot separately and thus creating new unique metadata where before it was shared. I got this idea from https://github.com/Zygo/bees/blob/master/docs/gotchas.md#snapshots:

If a subvol is deduped after a snapshot is created, the origin and snapshot subvols must be deduplicated separately. In the worst
case, this will double the amount of reading the bees scanner must perform, and will also double the amount of btrfs metadata
used for the snapshot; however, the "worst case" is a dedupe hit rate of 1% or more, so a doubling of metadata size is certain for
all but the most unique data sets.

Now I feel like this is, once again, a result of me using 0.10 without the extent scan mode... I would imagine extent scan mode does much better than this.

I've done that to work around old btrfs metadata bug that damaged a filesystem. I did the steps in a different order:

Oh, going in reverse is a very neat idea. Thanks!

The problem with this plan is that the metadata size will inevitably grow again. A filesystem filled by plain rsync with snapshots and no dedupe is the smallest possible metadata case, but metadata will continue to grow as long as at least one subvol in a snapshot series is updated. So with this plan you might do a lot of IO, but achieve only a temporary reduction in metadata size.

You are adding many TiB more space to the filesystem, so there is room for some metadata expansion--though we probably want to avoid having it grow by a factor of 100 if we can, which could happen if there are many dedupes missed by previous scans.

I think it's the 0.10 vs 0.11 thing here, once again. I didn't mind my dedup metadata increase until I shot myself in the foot with the whole hash table thing and sent bees to dedupe every subvolume.

So while you're assuming 0.11 and talking about the normal increase of metadata (my original 60 GiB), I was unhappy with the 0.10 subvol dedup explosion to 180 (and potentially much more, it was done with only like 10% of subvolumes at that point...).

So, if I do go through with this process, I should end up back at 60 GiB, which will be good.

bees v0.11 scans faster than btrfs balance can relocate data, so you can simply leave bees running during balance, file resize shrink, device remove or replace, or automatic bg reclaim. bees will keep the hash table up to date in near real time, fitting bees's IO into the idle periods of balance's IO.

Yeah, it looks like my main problem was not upgrading before changing the hash table size.

Or patch the kernel to increase the compression batch size (see kdave/btrfs-progs#960 (comment) for details) so that compress-force doesn't chop up the data into tiny pieces.

That's an interesting idea for some more tinkering. Thanks! (This whole server project was mainly for fun, tinkering, and finding out how things I didn't use before work — my first time trying out btrfs, bees, bcache, LVM... Because of this, I did some extreme and unnecessary things, basically just to mess around and find out)

@kakra

I'm pretty sure some of these steps may break btrfs-send/receive if you're using that. But it looks like you're using rsync.

I'm only using send-receive for the (readonly) snapshots that are fully deduped by that point (bees finished crawl). And the only case of my send-receive is to duplicate some snapshots (like backups of the server's own OS drive...) to another machine for redundancy.

I believe it should be safe in that case.

Don't force compression, leave the default heuristics. Maybe even do not use very high compression levels to reduce the chance of compressing files which compress poorly anyways. You'll gain more from deduplication. A lot of tiny extents (due to compression) will increase the meta data overhead and counter the effects of compression.

When building this server, I mostly wanted to mess around with btrfs (and find out), so I just turned on pretty much everything. I ran benchmarks on various compression algos/levels, found one that my server's CPU could handle faster than the drive write speed, and went with "force" because why not (from the manual I've understood it as simply "try harder" mode, as it would try compressing the whole file to see whether it's worth it, rather than just the beginning). I wanted to see how much the compression saved me at some point in the future, and then decide whether to keep it or not.

I guess I'll get rid of it. Especially if I'll be going through the plan of undoing the subvolume metadata explosion (which is basically rewriting everything to a new place, anyway).

duperemove is an extent-based dupe finder. It will use a lot of RAM. Try using it selectively against contents which bees struggles with. It should catch the remaining extents which fdupes didn't find. Always use duperemove from the same base directory so the relative path to your data stays the same. It will accumulate files paths in the hash database over time, up to the point where it becomes incredibly slow and memory consuming. Start over with a clean hash file then.
Finally let bees run for some time again, see where it struggles and take note of some of the directories which show up often and for a long time in its progress file. Then repeat steps 2 and 3 for that data.

How do I tell when bees is struggling with some data?
...although I'll probably find out once I read up on 0.11 stuff.

Since you're using my kernel patches, I hope you're using dedicated SSD/NVMe for meta data, which will speed up your use case a lot.

Yes indeed. I have two SSDs for RAID1 metadata, HDDs are data only. Your patches allowed me to finally consistently saturate a gigabit channel with backups, rather than only see such speeds when a huge new file is sent over 😁 It was magic. Thanks!

---

Once again, thank you very much! I've learned quite a bit. Most importantly, I learned that I should upgrade to 0.11 (and read up on new features and changes) before doing anything else about this, as it looks like all my trouble came from 0.10 stuff. Which probably confused everyone here for no good reason. Sorry about that 😅

I tried to keep my responses organized, but I kept learning new things as I went on with writing them, so it is probably a mess. Sorry about that, too.

So, my general plan is:

1. Drop forced compression from the mount args
2. Read up on and upgrade to 0.11
3. Increase hash table further to 8 or 10 GiB to cover extreme edge cases of data ingress
4. After hash table is wiped, block bees from scanning older data via transid edits
5. Create an rw subvolume from the newest rsync state (let's call this subvolume tmp-migrate)
6. For each existing old snapshot (going in reverse), rsync old snapshot to tmp-migrate, wait for bees to dedup, delete old snapshot and replace it with a fresh one from tmp-migrate

And after that I should be good with metadata back to reasonable size. (60 GiB or even less without the forced compression.) And in the future, I should:

• Keep bees running as I rsync (if I feel paranoid about it affecting bandwidth, I can always benchmark)
• Keep bees running as I balance
• Upgrade sooner 😬

[Zygo]

You're talking about --inplace specifically, right? How does it hurt rsync over slow networks?

It prevents sparse file handling and disallows copies that refer to later data in the file (i.e. an existing block can only be reused if it is before the currently transferred file position or exactly at that position). Without --inplace, a delta can refer to any block within the existing file on the receiver.

If your workload is primarily adding new files or appending to existing ones, there is not much difference, but if it's VM images then you can lose some efficiency. A VM image is more likely to contain new copies of existing data at a lower offset, but with --inplace, rsync can't reuse the copy at the higher offset, so it will send the data again.

My idea here was to reduce hard drive random access (deduping one file while writing another one intuitively sounds like a recipe for wrecking throughput).

Running bees with idle priority (ionice -c3) helps here, or you can enable loadavg tracking in bees so that bees backs off when disks get saturated.

You can also get some benefit from caching if bees is reading data that's still warm in the page cache after rsync wrote it. It's not ideal due to kernel limitations on the page cache, but it's not zero either.

If you have metadata on SSD, then the cost of bees metadata searches will be negligible. bees and rsync will fight for the HDD heads only for the data blocks. rsync won't saturate spinning disks for writing with only a gigabit feeding it--though rsync does read data for copying, which might slow the disks down a lot.

Dedupe is a pretty big IO latency hit, even on SSD, because it forces btrfs to lock its trees. rsync will be slower if you have heavy duplication on large files since write and dedupe operations exclude each other, and dedupe forces an immediate fsync-like flush of the file's data which might not be optimal for a spinning drive.

The total rsync & bees (concurrent) time should still be less than the rsync + bees (sequential) time, as there's considerable idle IO time in both workloads. If you need to absolutely minimize rsync time (e.g. due to limited network availability or a scheduled maintenance window), then pausing and resuming bees with SIGUSR1 and SIGUSR2 during the rsync might be the only way to go.

My server is still on v0.10, huh. Time to upgrade.

v0.10 and earlier have no limits on fragmentation and poor handling of closely related references (like snapshots), so they will absolutely explode the metadata on a high-snapshot workload.

v0.11 still has some metadata expansion with snapshots, but it's much closer to the theoretical minimum. v0.11 also has some improvements for IO concurrency on spinning drives.

The upgrade is fairly simple--make sure you don't have a -m option in your configuration (the default is the new extent scan). If you're resizing the hash table at the same time then there's nothing else to do.

How do I tell when bees is struggling with some data?

Enable the status output, which will report what all the threads are doing once per second. If you see the same file over and over in dedupe operations, that's what bees is struggling with.

Alternatively, enable the bees 'info' level log output (-v7), which reports every deduped extent.

[kakra]

@Watkurem

Removing the images and making fresh copies with nodatacow resulted in proper 150-200 MiB/s speeds that the hard drive is capable of.

I had a similar observation but later I found out that even with nodatacow, the images may split into thousands of extents just because you take snapshots every now and then, i.e. for backups. What worked better for me is using qcow2 images with a chunk size of 2 MB, and disabling compression on the directory (before creating the images) with chattr +m. This way, the images are also checksum-protected by btrfs. I never had good VM performance with nodatacow. Especially compression seems to be what's causing millions of small extents, it just happens that nodatacow also disables compression. Additionally, bees won't dedup nodatacow files (due to missing checksums). So if you transfer that attribute to your backup, too, there won't be compression, and there won't be dedup for those files.

I wish that btrfs had a special feature to create virtual block devices like zfs has with vdevs. That way, btrfs could optimize access and write patterns especially for such virtual block devices outside of regular file access patterns (with all their posix guarantees and features) and we could have fast virtual "raw" block devices for qemu.

As there's no point in a backup that I cannot read in reasonable time frame, I opted for not keeping history (and checksums) of the VM images in favour of overall speed.

I'm not using btrfs as a backup target especially for that reason, and because bugs in btrfs destroyed my backup target multiple times in the past (but that's probably almost 10 years ago, it has been rock solid for a long time now). Instead, I'm using borg or restic on classic xfs. It has way superior dedup and compression, is very fast during backup, and also fast during restore. Accessing single files and folders from the backup vault is possible using fuse - but that's one operation which of course is a lot slower than bare btrfs access.

How do I tell when bees is struggling with some data?
...although I'll probably find out once I read up on 0.11 stuff.

Yeah, v0.11 has a useful dashboard-like run state file with progress tables, and a list of current tasks. I think, this task list was already present in v0.10 but task handling has also improved. I'm watching this file during operation to see repeated patterns of accessing the same files over and over again over a long period of time. Usually, this distills down to a folder structure which you'll know bees will struggle with the whole time. Removing such files from the old transids by rewriting them can speed up bees progress considerably (but only if you manage to completely purge them from all sharing snapshots). If you rewrite all your snapshots while bees is running, this result will occur naturally. Otherwise you can use btrfs-defrag, fdupes and duperemove to especially target such directories across multiple snapshots, or rewrite to a temporary copy, then use fdupes or duperemove.

For each existing old snapshot (going in reverse), rsync old snapshot to tmp-migrate, wait for bees to dedup, delete old snapshot and replace it with a fresh one from tmp-migrate

Oh yeah, that's smart. Use a scratch directory to sync to the next older snapshot, then snapshot that scratch directory. This introduces slightly different write patterns which bees has to deal with, so @Zygo may say how smart that really is. :-D

That's actually how I did my backups to btrfs targets when I still did that: using a scratch directory as the backup target, then snapshot that directory if rsync successfully finished.

[Watkurem]

@Zygo

It prevents sparse file handling and disallows copies that refer to later data in the file (i.e. an existing block can only be reused if it is before the currently transferred file position or exactly at that position). Without --inplace, a delta can refer to any block within the existing file on the receiver.

If your workload is primarily adding new files or appending to existing ones, there is not much difference, but if it's VM images then you can lose some efficiency. A VM image is more likely to contain new copies of existing data at a lower offset, but with --inplace, rsync can't reuse the copy at the higher offset, so it will send the data again.

Makes sense. Thanks!

Running bees with idle priority (ionice -c3) helps here, or you can enable loadavg tracking in bees so that bees backs off when disks get saturated.

You can also get some benefit from caching if bees is reading data that's still warm in the page cache after rsync wrote it. It's not ideal due to kernel limitations on the page cache, but it's not zero either.

I'll benchmark and look into the priority options if I'm not happy with the results. But considering all I've learned so far, I probably won't even notice bees running alongside my rsync.

My writethrough bcache will also probably help with this.

The total rsync & bees (concurrent) time should still be less than the rsync + bees (sequential) time, as there's considerable idle IO time in both workloads. If you need to absolutely minimize rsync time (e.g. due to limited network availability or a scheduled maintenance window), then pausing and resuming bees with SIGUSR1 and SIGUSR2 during the rsync might be the only way to go.

Yeah, I don't really mind bees taking a long time to dedup, because this is a once-per-week thing and the server doesn't do much else. So even if it took, like, five days, I'd be fine with it.

But the rsync itself taking much longer would be a minor annoyance. Anyway, I'll see if it feels slow to run bees alongside the rsync, and benchmark the difference if it does. It seems like a bigger hash table would allow me to do the pause-and-resume thing without any issues.

The upgrade is fairly simple--make sure you don't have a -m option in your configuration (the default is the new extent scan). If you're resizing the hash table at the same time then there's nothing else to do.

Yep, I was using the default. Thanks!

@kakra

Especially compression seems to be what's causing millions of small extents, it just happens that nodatacow also disables compression.

In hindsight, that could very well be the reason. I was comparing with the machine that uses the images, but it doesn't have neither dedup w/ windows files, nor compression, so it was always both on or both off.

I'm not using btrfs as a backup target especially for that reason, and because bugs in btrfs destroyed my backup target multiple times in the past (but that's probably almost 10 years ago, it has been rock solid for a long time now). Instead, I'm using borg or restic on classic xfs. It has way superior dedup and compression, is very fast during backup, and also fast during restore. Accessing single files and folders from the backup vault is possible using fuse - but that's one operation which of course is a lot slower than bare btrfs access.

I specifically wanted to try out btrfs, and the backup scenario seemed like a no-brainer for checking out CoW features at scale. I'm pretty happy with how it works. I am also a big fan of filesystems spanning multiple devices, so it was btrfs or zfs, and zfs is less flexible (can't be grown with random drives). bcachefs looks very promising, but I didn't know about it at that point, and AFAIK it's not particularly stable yet.

I only had a few data-eating spooks with my setup, but all were recoverable and caused by bcache and power shenanigans, none by btrfs itself. Then again, I never did particularly stress this setup; rsync and bees is all it does.

Oh yeah, that's smart. Use a scratch directory to sync to the next older snapshot, then snapshot that scratch directory. This introduces slightly different write patterns which bees has to deal with, so @Zygo may say how smart that really is. :-D

That's actually how I did my backups to btrfs targets when I still did that: using a scratch directory as the backup target, then snapshot that directory if rsync successfully finished.

I thought that's exactly what you suggested — a temporary subvolume on the existing filesystem 🤔

It seems like it should work just fine, as all writes will be "new" to bees (since it won't look at previous transid and hash table will be blank). But I will need to make sure that rsync will not use reflink cp, or it won't work out.

---

Ok, thank you both very much once again! I'll make some time to upgrade the server and set up this recovery procedure, and let you know how well it works. Hopefully, this issue will help someone else in the future.

[Watkurem]

Oh, another thing I do is keeping BEESHOME on a different drive (SSD). I remember reading somewhere that it's theoretically a good idea, and my beesd script is edited to allow this:

if [ ! -d "$BEESHOME" ]; then
    INFO "Create subvol $BEESHOME for store bees data"
    btrfs sub cre "$BEESHOME"
#else
#    btrfs sub show "$BEESHOME" &> /dev/null || ERRO "$BEESHOME MUST BE A SUBVOL!"
fi

Is it still a good idea? To avoid competition for the drive heads and such.

There's a still open #272, and the check seems to be gone from the new beesd script:

bees/scripts/beesd.in

Lines 121 to 124 in ba11d73

	if [ ! -d "$BEESHOME" ]; then
	INFO "Create subvol $BEESHOME for store bees data"
	btrfs sub cre "$BEESHOME"
	fi

so I can only assume this is fine.

[Zygo]

Use a scratch directory to sync to the next older snapshot, then snapshot that scratch directory. This introduces slightly different write patterns which bees has to deal with, so @Zygo may say how smart that really is. :-D

It doesn't matter if you do "rsync to A, snap A B, rsync to B, snap B C, rsync to C" or "snap tmp A, rsync B to tmp, snap tmp B, rsync C to tmp, snap tmp C...delete tmp". Either way, the snapshot creates a second reference to the thing that was just rsynced, and the next rsync triggers metadata conversion from shared pages to reflinks. Metadata sharing is symmetrical, so modifying any subvol that is sharing metadata with another subvol causes the same metadata growth and IO work.

A variation on this is to keep the "tmp" snapshot around permanently, and use it as the target for future rsyncs, making a snapshot of it whenever the rsync completes successfully. Same total number of operations, same metadata IO pattern, only the filenames and some ID numbers in the metadata pages are different.

The key optimization is the reverse chronological order: "truncate new files to make old ones" is better than "append to old files to make new ones." There's over 100 bytes of metadata for every fragment of data, but a reference to any part of that data is only about 60 bytes. Writing two versions of a growing file in forward chronological order produces a minimum of 260 bytes of metadata:

• 100 bytes for the whole first version of the file,
• a 60-byte reference to the first part in the second version of the file,
• and another 100 bytes for the second part of the second version.

Reverse chronological order produces a minimum of 160 bytes:

• 100 bytes for the whole second version of the file
• 60 bytes for a partial reference to only the first part in the first version of the file.

These aren't exact numbers. They include average affects on csum and free space trees.

keeping BEESHOME on a different drive (SSD). Is it still a good idea? To avoid competition for the drive heads and such.

There's not very much competition--during normal operation, the IO rate for the hash table is limited to 128KiB/second and the crawl table is written once every 15 minutes, so it's only "competition" if you're running bees on a cheap SD card that maxes out at sub-MB speeds.

Back in the day, you could put BEESHOME on its own subvol or on another filesystem if you wanted to avoid old btrfs bugs (well, not that old--the last one was fixed in 2022). It's still a good idea to use a subvolume if BEESHOME is on the same filesystem, simply to avoid including it in backups (it's useless on a restored filesystem because all the physical data locations will be different). If BEESHOME is on another filesystem--even a btrfs one--then subvolumes are irrelevant.

The subvol check in v0.10 and earlier was removed in v0.11, but the automatic setup (which assumes the same btrfs filesystem) still creates a subvol if you didn't create a directory yourself.

There was another issue about BEESHOME on non-btrfs (particularly on xfs) that was fixed in v0.11. Now you can put BEESHOME anywhere you like (well, maybe not filesystems with 8.3 filename limits, but certainly anywhere else).

272 is still open for the discussion about how and when to create BEESHOME objects automatically, especially when on other filesystems. I suppose 272 could be closed and the design notes can just sit on my TODO list.

[kakra]

My writethrough bcache will also probably help with this.

Consider writearound: You don't want data written only once to end up in the cache, especially if your rewrite all your snapshots. Writes are mostly linear anyways, thus they are fast (especially combined with my dedicated-metadata patches), and most reads will be random because they are not in the cache, and are mostly likely not read after being written.

bcachefs looks very promising, but I didn't know about it at that point, and AFAIK it's not particularly stable yet.

Just my 2 cents: Yeah, it looks promising, and probably is technically in a good state already. But I have some concerns with it: The author isn't able to properly communicate with kernel upstream, so it will most likely stay out of tree in the future (like zfs for the "out of tree" part, just for other reasons). The author is not acting very professionally by bashing every other file system and their developers and their abilities, especially btrfs. And the author tends to abandon projects for new ones (he abandoned bcache to start bcachefs) - who tells us this won't be happening with bcachefs? Also, I'm reading more posts about bcachefs corruption and data loss than about such issues with btrfs in the last few years. But I give him the point that bcachefs probably fixes bugs faster than btrfs. And I think from a technical view, the author is sort of genius.

Oh, another thing I do is keeping BEESHOME on a different drive (SSD).

Yeah, I have it on xfs hosted on an SSD where possible. But I also have systems where it lives right in a dedicated subvol of the root subvol. It works fine for both, but systems with dedicated beeshome tend to be snappier (OTOH, those systems are usually more capable anyways). I think there's not much difference: Writing is rate limited anyways (as @Zygo wrote), and upon start, the file will be read into memory, memlocked, and changes will be slowly written back to disk. So it doesn't even read from the file to lookup hashes.

We once has a discussion if updating an in-filesystem hash table and bees reading new transids from that same filesystem could generate a feedback loop which never ends. Not sure how that ended up, I don't think we had a final conclusion about this.

There was another issue about BEESHOME on non-btrfs (particularly on xfs) that was fixed in v0.11.

Oh well, that may explain some hiccups I experienced in the past... :-D Good it's fixed now.

[kakra]

@Watkurem BTW: If you are using my patchset, consider using and configuring the queue-balancing patches, too. Since you are using a multi device raid1 setup, this should greatly improve your proposed rewriting procedure as our benchmarks have shown. It doesn't introduce any incompatible on-disk changes and can be changed on the fly, so you can adjust it for your workload. But the queue strategy seems to work best overall.

This would be a nice opportunity to benchmark "rsync from and to the same pool". :-)

[Zygo]

We once has a discussion if updating an in-filesystem hash table and bees reading new transids from that same filesystem could generate a feedback loop which never ends.

bees identifies the hash table inode and excludes it from dedupe, so there's no feedback that way. There is some feedback from writing the log to the filesystem, but that's legitimate new data.

beescrawl.dat does generate a feedback loop since updating that file must increment the filesystem transaction counter, and the file must be updated when the transaction counter is incremented--but that loop does one extent of IO every 15 minutes, so it's not exactly burning SSDs with write storms.

Adding hysteresis to the scanner (so they don't start dedupe until they get N transactions behind the filesystem's current transid) could block that loop. If N is 10 or 100, then one or two transactions can't wake up the scanner any more, and bees would no longer wake itself up with its own checkpoint updates. There would still be feedback, in the sense that bees will scan its own checkpoint file in the middle of other data, but no loop when it's otherwise idle. (and now that I think about it, the checkpoint file can be explicitly excluded too)

The big looping problem back in the day was that bees would try to dedupe across extent boundaries while splitting extents. At some point there would be a pair of adjacent runs of identical extents with different internal boundaries, and every pass of the scanner would try to do a 3-way dedupe with extents A and B deduping C, then B and C deduping A, then C and A deduping B, repeat forever (or some far longer chain that couldn't be broken by storing a list of bad extents in memory). The temporary fix to stop the looping was to simply never dedupe across the existing extent boundaries. The permanent fix is to design a dedupe engine that can detect and break those loops before touching the filesystem, and replace bees's dedupe engine with that new code.

Not sure how that ended up, I don't think we had a final conclusion about this.

Technically, there's a workaround in place, but it's still open. The temporary fix is a significant performance hit, but it stops the loops. The permanent fix is to figure out how to ensure that the output of the new dedupe engine never gets through its input filter.

[Watkurem]

Still haven't had the time to work on the recovery, but I did remember why I put the beeshome on a separate drive. It was actually much simpler than trying to avoid competition for the heads!

Since my server's backup storage sits idle most of the time (unless I'm rsyncing to it, or it's actively deduping), and the drives are loud and kind of power hungry, I configured them to go into standby (Z state as seachest refers to it) after 30 minutes of inactivity. However, bees would wake them up all the time to make an update to the state files, or something like that. Moving beeshome to the root SSD resolved this