Skip to content

oans v1.0.0

Choose a tag to compare

View all tags
@martinus martinus released this 17 Jul 05:37
v1.0.0
a28e865
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

oans is a friendly, faster fork of duperemove for btrfs & xfs deduplication. All of the original design and code is the work of Mark Fasheh and the upstream contributors; this is the first tagged release of the fork, breaking from the inherited 0.15.x line so versions no longer clash with upstream.

Drop-in: the oans binary installs a duperemove compatibility symlink, and the hashfile format and CLI are unchanged from upstream.

Highlights

  • Skip already-shared files — files whose extents are already shared are detected up front and skipped. On an already-deduped tree (~2M files, ~230 GiB on btrfs): ~90s vs ~11m for upstream 0.15.2.
  • No cross-generation reprocessing — large duplicate groups spanning many dedupe passes are handled once instead of re-checked each pass: ~36% faster on a real 2M-file tree, with accurate dedup accounting.
  • Batched hashfile transactions — collapses a per-file SQLite lock storm (hundreds of thousands of F_SETLK → a few hundred); ~24% faster rescans.
  • Parallel directory walk (--io-threads), with a default tuned to where btrfs metadata contention plateaus.
  • Compact path-hash index for faster path lookups on large trees.
  • Hardlink safety fix — guards an INSERT OR REPLACE cascade that could silently empty the hashfile (with a regression test).
  • Clearer output — human-readable colorful summary and a live dedupe progress bar with throughput/ETA.
  • Tests & CI — a Python integration suite (57 tests) driving the built binary against a scratch tree, plus GitHub Actions.

Benchmarks are real but workload-dependent (the already-deduped case is the skip's best case). On compressed filesystems, judge reclaimed space with compsize Disk Usage, not the logical "Deduplicated" figure.

Install

make && sudo make install    # installs oans + a duperemove compat symlink

Requires glib2, sqlite3, libxxhash ≥ 0.8, util-linux, libbsd; Linux kernel ≥ 3.13.

Credits

Original author: Mark Fasheh and the upstream duperemove contributors — https://github.com/markfasheh/duperemove. Licensed GPLv2.

Assets 2
Loading