_Much_ faster now.
Highlights
----------
This release introduces *extent scan mode* and other improvements for dramatically increased dedupe performance, particularly on VM images and snapshot-heavy workloads. Users can expect substantially faster scanning and free space recovery, especially on filesystems with many snapshots or reflinks.
This release takes advantage of newer kernel features than previous bees releases. Users running kernels older than 5.4 LTS or 5.7 (i.e. released before May 2020) should carefully assess the risks of running kernels with known bugs.
*Subvol scan mode* (the only scan mode supported by bees releases up to v0.10) is now deprecated, and may be removed in a future release. All users are strongly encouraged to switch to extent scan mode. A seamless transition without rescanning old data is supported when previous subvol scans are up to date.
Major New Feature: Extent Scan Mode
-----------------------------------
* Extent scan mode scans the extent tree directly, instead of scanning through all the subvol trees separately. It is the new default scan mode.
* Each extent is scanned only once, eliminating redundant reads and avoiding rescanning of shared extents across subvols and snapshots. This significantly improves performance on filesystems with existing snapshots or reflinks.
* Largest extents are processed first for fastest free space recovery.
* Cleans up temporary extents immediately to minimize space usage during dedupe.
* Transition from subvol to extent scan mode is straightforward: if all subvol scans are complete, extent scan will process only new data created in subsequent btrfs transactions, without rescanning existing data. If some subvol scans are incomplete, extent scan can rescan the entire filesystem much faster than subvol scan does.
Performance Improvements
------------------------
* Progress reporting (extent scan mode only): shows estimated data size and expected time to finish scanning in logs and status outputs.
* Smarter dedupe criteria: prevents dedupe operations that save less than half the blocks in an extent or require more than 100 copy/dedupe operations to free space. This avoids excessive fragmentation, especially in large files with small updates, such as databases and VM images.
* Adds dynamic rate throttling: the new `--throttle-factor` option allows bees to estimate the rate at which the kernel deletes extents and slow down dedupe requests to match this rate. This experimental feature is intended to prevent the kernel from accumulating a large backlog of extent deletions in memory during a btrfs transaction. Tuning this may help avoid filesystem write stalls on systems with heavy delete workloads.
* Better concurrency: threads now dynamically redistribute their work so that they can make continuous progress without waiting for each other. This results in more efficient parallel dedupe.
* Removed excessively costly "toxic extent" workarounds: no longer needed in current kernels, yielding up to 100x–1000x speed boost on some workloads.
* Better I/O scheduling:
* Coordinates between threads to prevent thrashing during disk reads.
* Prefetches data into page-cache before dedupe calls.
* Submits full-extent dedupe operations, avoiding obsolete 16M kernel limits.
* Extent refs cap lowered from about 700,000 to 9,999:
* Reduces the size of `ioctl` buffers, which avoids forcing the kernel to evict pages from memory to accommodate large buffers.
* Limits reference counts for very frequently duplicated data, which avoids performance issues in btrfs (even when bees is not running).
Security Improvements
---------------------
* Uses the new `openat2` system call for improved robustness against rename and symlink attacks. Currently, `openat2` support in the kernel is optional, and bees will fall back to `openat` if `openat2` is not available.
Note: In a future release, the fallback will be removed, and `openat2` will become mandatory.
* `beesd` now uses a private mount namespace for isolation even without `systemd`.
Operational Improvements
------------------------
* Warning and debug log verbosity reduced.
* `SIGUSR1` and `SIGUSR2` signals allow bees to be paused and unpaused without preventing bees from closing open files and subvols during the pause. Previously, pausing bees was only possible with freezer cgroups or with `SIGSTOP` and `SIGCONT` signals, which could prevent entire subvols from being deleted during the pause.
Compatibility Improvements
--------------------------
* It is now safe to locate `$BEESHOME` on filesystems such as XFS that lack data flushing on `rename`.
* Fixed 32-bit `ioctl` compatibility for mixed 32/64-bit systems.
* bees will now detect a `btrfs send` in progress, and wait for it to complete automatically. `--workaround-btrfs-send` is no longer necessary when used with extent scan mode.
New Kernel Requirements
-----------------------
The minimum recommended kernel version has been raised to 5.7 or 5.4 LTS. On older kernels, bees may trigger a kernel crash, hang, or data corruption bug. *Kernels older than the oldest active LTS kernel will never be fixed—use at your own risk!*
* **4.15** is the oldest kernel that supports extent scan.
* **5.2** is the oldest kernel that supports dedupe and send at the same time without a workaround.
* **5.6** is the oldest kernel that supports `openat2`.
* **5.7** or **5.4 LTS** is the oldest kernel that does not require toxic extent workarounds.