oans v1.6.0

Fixes a bug that silently excluded sparse files from deduplication, and changes
--exclude to .gitignore pattern syntax.

Fixes

• Sparse files whose size is not a multiple of the block size were silently
  skipped, on every run (#152). They were reported as having changed while
  being hashed — which was not true — and then dropped, so they never reached
  the hashfile and were never deduplicated. The tail of a trailing hole was
  rounded down to a block boundary, leaving a final partial block that nothing
  could read; the scan then concluded the file had changed under it and threw
  the digest away. Nothing was stored, so the next run repeated it, forever.
  This hit exactly what sparseness is used for: VM images, database files and
  preallocated media. If you have such files, this release will hash and
  deduplicate them for the first time.
• Three different conditions all reported the word "changed", with three
  different meanings and no numbers (#152). Each now states what was observed,
  the likely cause, and what follows from it — the scan-phase one in particular
  is routine on files that are still being written and now says so.
• The dedupe summary no longer reports the wrong cause. One counter was
  incremented both by our own pre-flight size check and by the kernel's
  byte-compare verdict, then printed as a single "changed since scan" figure —
  so every kernel mismatch was attributed to a file change that had not
  happened. The two are now counted and named apart.

Changes

• --exclude now uses .gitignore pattern syntax (#154, closes #147), the
  same syntax git, ripgrep and fd use:

  Pattern	Meaning
  @eaDir, *.iso	no / — matches the name at any depth
  /srv/media/cache*	leading / — absolute, anchored
  Steam/temp	interior / — matches at any depth
  cache/	trailing / — directories only
  * ? [a-z] **	* stops at /, ** crosses

  Previously patterns were matched with fnmatch() against the whole path, and
  a pattern without a leading / was resolved against the current directory —
  so --exclude node_modules matched at most one literal directory, and usually
  nothing at all, without saying so. The obvious things to write (@eaDir,
  .snapshots, node_modules) were all silent no-ops.

  This is a breaking change. Patterns already stored in a hashfile are
  replayed under the new rules, so a scheduled job may now exclude more, or
  less, than it did. Run your job once by hand and check the output before
  relying on the next timer firing. Two things make that easier: a pattern
  matching nothing is now reported as a warning (including under --quiet, so
  it reaches the journal), and a malformed pattern is an error before the scan
  starts rather than something the run ignores while scanning a wider tree.

  Negation (!) is not supported.

• --autotune has been removed (#153, closes #150). It measured hashing
  throughput at several thread counts and stored the winner in the hashfile.
  Without root it cannot drop the page cache, so it measured memory bandwidth
  instead of disk — which keeps scaling with threads well past the point where
  btrfs metadata contention caps real throughput. It printed a warning and
  persisted the answer anyway, into the hashfile a systemd timer then replays
  forever. On a 32-core NVMe machine it recommended and stored 16 threads where
  the measured plateau, and the built-in default, is 8. --io-threads is still
  sized automatically from the detected storage, and can still be set by hand.

Performance

• Exclude matching is now independent of pattern count. Patterns compile to
  a single combined PCRE2 regex with JIT enabled, rather than one fnmatch()
  per pattern per file. Measured on a 250k-file tree: ~160 ns more per file in
  the worst case (warm cache, a pattern active), around 0.5% of wall time on the
  most walk-bound workload. GLib was already a dependency, so this adds none.

Documentation

• The README is up to date with 1.4.0 and 1.5.0, and a stale hashfile-size
  figure that appeared in no benchmark has been corrected (#151).
• The NAS quick-start lost its --autotune step, and gained the upgrade note
  for the --exclude change.

---

Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.

oans v1.5.1

A patch release for three problems that shipped in v1.5.0: a hang, a
quadratic scan on fragmented files, and a progress display that could read
like a hang of its own.

Fixes

• Fixed a hang at the end of the scan phase (#138). The scan-phase
  progress thread decided when to stop by comparing running sums against
  totals, so any file counted into a total but never credited back left the
  two unable to ever meet — the thread spun in usleep forever and
  pscan_join() blocked for good, with the scan finished and every worker
  already exited. The scan phase now takes the same producer-driven shutdown
  the dedupe phase already had: only the producer knows when the work is
  over, so only the producer ends the thread. It takes a small machine to
  hit — 12 copies of the progress tests pinned to 2 cores hang ~15% of the
  time, the same load unpinned on 32 cores never does — which is why it
  first read as CI flakiness.
• Hashing workers no longer sit on a stale "commit" line when the walk is
  the bottleneck (#143). A csum worker keeps its progress slot for its
  whole life and rolls it from file to file, so between files the line kept
  whatever the last file left behind. Invisible on a hashing-bound run; on a
  big NAS tree where only a handful of files need hashing, all four workers
  showed commit <last file> for the rest of the run, reading like a hang in
  the commit path. A slot now parks as idle once its worker has waited 250 ms
  for work — long enough that the sub-millisecond gaps between small files
  never flicker.

Performance

• Fixed a quadratic extent scan on fragmented files (#134).
  is_area_ignored() called get_extent() with a NULL resume cursor from
  inside its own loop, so every iteration rescanned the extent array from
  index 0 — the cursor exists precisely to avoid this, but the hottest caller
  never passed one. It is reached once per 1 MiB on every scan, and per block
  under --dedupe-options=partial, so the cost scales with extents per file.
  On a 1 GiB file with 65536 extents, get_extent was 48.93% of all CPU;
  threading the caller's cursor through drops it off the profile entirely and
  halves the wall clock (median 1.76s → 0.88s). Ordinary source trees have
  few extents per file and are unchanged — but dedupe itself fragments, so
  this hit exactly the re-run workload oans targets. Output is byte-identical.

CI & tooling

• The integration suite runs in parallel (#139): 26.4s → 7.9s on 4 cores,
  and the sanitizer legs that re-run all of it at 4–8× the cost benefit most.
• Every CI job is bounded by a timeout (#135), after a wedged TSAN leg read
  as "in progress" for 23 minutes; tests/run.py also warns when run without
  ThreadSanitizer suppressions, which otherwise fails ~85 tests as pure artefact.
• apt packages are cached between runs (#141), most of the wall time on the
  short legs and ~14% of the valgrind leg that sets CI's critical path.
• The TSAN pool-lifetime annotation keys on a token, not the pool pointer
  (#140) — the pointer is exactly what teardown clears, so reading it to
  publish on was itself a race.
• Three silent tooling failures are now loud (#137), and the benchmark
  harness gained a fragmented profile (#136) — the only one with real
  extent counts, so a reintroduction of #134 would be caught.
• A hardlink test no longer asserts an exact extent count (#142); how many
  extents a file gets is btrfs's business.

---

Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.

oans v1.5.0

Reaches files that were previously skipped, fixes two crashes, and puts the
whole codebase under ASAN, UBSAN and ThreadSanitizer in CI.

Features

• Hash and deduplicate files whose absolute path exceeds PATH_MAX (#124,
  closes #117). Previously these were skipped with a warning. The kernel rejects
  any single pathname argument over 4096 bytes, so the new src/longpath.c
  reaches them by opening a reachable ancestor and openat-walking the rest,
  advancing by the longest run of components that fits one syscall argument — a
  4620-byte path costs 5 opens instead of 22. In-range paths take the plain
  open()/opendir()/stat() fast path internally, so the hot path is
  unchanged. No hashfile schema change and no DB_FILE_MINOR bump.
• Hashing throughput during the scanning phase (#120). The progress display
  now shows live MB/s while hashing, not just during dedupe.

Fixes

• Fixed a use-after-free in partial mode (#128, fixes #123) that caused
  intermittent crashes, plus three remaining PATH_MAX assumptions.
• Fixed a use-after-free found by ThreadSanitizer (#129).
• The version object now rebuilds when git describe changes (#122), so a
  binary can no longer report a stale version string.

Hardening

• CI gained clang ASAN and UBSAN legs running both test suites (#125), and
  now fails on warnings with stricter flags plus dependabot (#127).
• ThreadSanitizer is usable against GLib (#129), with a suppression that
  matches current distros (#131).
• CI runs once per commit instead of twice (#130), and make check-all
  runs every CI leg in one command (#132).

All of this runs against a real btrfs and XFS scratch filesystem.

Tooling

• scripts/bench-dedupe.py (#121), a reproducible larger-than-RAM dedupe
  benchmark harness; methodology is documented in docs/benchmarks.md.

---

Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.

oans v1.4.0

oans 1.4.0 is a large feature release focused on the dedupe phase, a
unified live progress UI, and honest observability — all backward
compatible with 1.3.0 (same CLI superset, same hashfile schema, hashfiles
carry over).

Highlights

• Fast dedupe on trees larger than RAM (~13× vs duperemove 0.15.2). When the
  working set doesn't fit in the page cache, the kernel's FIDEDUPERANGE byte
  compare re-reads cold. oans now keeps just-hashed data warm and prefetches each
  dedupe round as fast sequential I/O (#107). Median hash+dedupe on a
  cache-capped ~10.5 GiB tree: 13.8 s vs 179.7 s, at ~2× lower RSS and a
  ~1.8× smaller hashfile — byte-for-byte identical sharing. See
  docs/benchmarks.md.
• Streaming dedupe pipeline. The dedupe phase runs one persistent worker pool
  with a bounded double-buffered producer, so peak RSS stays low and flat
  regardless of how many duplicate groups are in flight (#111, #112, #116).
• Unified live progress UI across scanning / hashing / dedupe / done, with a
  byte-weighted smooth dedupe bar and a weighted-progress scan ETA.

Dedupe

• Byte-weighted, smooth dedupe progress bar — moves 0→100% by kernel byte-verify
  volume, even through one giant group (#111).
• Order-independent extent loader; whole-file dup members excluded via a
  correlated probe (#112).
• Streaming pipeline with a persistent pool and generation-ordered watermark,
  preserving the Ctrl+C safety invariant (#116).
• Keep hashed data in the page cache for the dedupe phase and prefetch each
  FIDEDUPERANGE round (#107).

Scan & progress

• Weighted-progress scan ETA (#98); retuned HDD per-file weight (#101);
  "mapping:" phase shown during pre-read setup (#99).
• Hash largest files first (LPT scheduling) to shrink the idle tail (#91).
• Skip all-hole blocks in sparse files (#87, #89).
• Refuse unsupported-fs roots up front instead of failing silently (#97); XFS
  identified without root, with a loud failure when it can't (#85).
• Warn instead of silently skipping paths over PATH_MAX (#115).
• Grow hash arrays geometrically (#95); block-aligned block-count estimates (#94).
• Scan diagnostics: contention/starvation counters, honest small-file ETA, no
  more idle-flicker (#93).

Observability & reporting

• --progress=json: machine-readable per-phase progress for dashboards.
• --stats relabels "reclaimable" as "duplicated" (a logical figure, not a
  to-do) (#84).
• Self-contained --help/usage instead of shelling out to man (#86).

Reliability & CI

• Fix a use-after-free and leak when recreating a rejected hashfile (#105).
• CI runs the unit + integration suites under valgrind, on btrfs and XFS (#106).

Docs & tooling

• Consolidated all benchmark information into a single
  docs/benchmarks.md
  with refreshed larger-than-RAM numbers (#109, #118).
• Consolidated benchmarking into scripts/bench.py (#102).
• README restructure of "What the fork changes" (#110); note CI covers XFS (#96);
  Fedora 41 libatomic dependency (#82).

oans v1.3.0

oans 1.3.0 — lower memory use, prebuilt binaries, and polish.

Performance — lower peak memory

• Smaller per-read-thread hashing buffers and per-role SQLite page-cache budgets cut peak RSS substantially on large trees (e.g. ~70 → ~15 MiB on large-file scans at --io-threads=8), with no measurable change in scan/dedupe speed. (#80)
• The per-scan hardlink-guard set is now a compact open-addressing set instead of a GHashTable, roughly halving its per-file overhead on multi-million-file scans. (#81)

Distribution

• Prebuilt x86_64 Linux binaries are now attached to every release automatically — grab oans-1.3.0-linux-x86_64.tar.gz below. (#75)
• In-tree AUR PKGBUILDs for oans (release) and oans-git. (#74)

UX

• The "Compacting the rebuilt hashfile …" line now completes with "done", so a finished run no longer looks like it is still working. (#77)

Docs & project

• Complete, accurate man-page rewrite. (#73)
• README: "How it compares" (vs bees / ZFS / duperemove), a benchmark-methodology page, a demo screencast, and a logo. (#74, #76, #78)

Full changelog: v1.2.0...v1.3.0

oans v1.2.0

The first release carrying the fork's full feature set — 31 commits since 1.1.1. Everything is additive and backward-compatible; the CLI is a superset of duperemove's and hashfiles rebuild automatically.

✨ New features

• --stats — inspect a hashfile: file/hash counts, duplication ratio, file-size summary, the top duplicate groups with an example path, and how much a VACUUM would reclaim.
• Run history & metrics — every run is recorded in the hashfile. --history shows space reclaimed over time and a lifetime total; --json exports current metrics for a dashboard.
• Self-describing hashfile — each run stores its options, paths and excludes, so oans --hashfile=FILE (no other arguments) replays the last run incrementally. Refuses rather than pruning if every stored path has vanished.
• --autotune — benchmarks your backing storage (NAS/HDD/RAID) and picks the fastest --io-threads, persisting the winner in the hashfile.
• systemd timer templates — make install-systemd, then systemctl enable --now oans@<name>.timer for weekly, idle-priority dedupe. See systemd/.
• NAS quick-start guide — a complete walkthrough from first scan to scheduled, monitored dedupe: docs/nas-quickstart.md.

⚡ Performance

• Skip already-shared files up front instead of re-reading them — warm re-runs on a mostly-stable tree are dramatically faster (a deduped 2M-file / 230 GiB tree rescans in ~92 s vs ~11 min for upstream 0.15.2).
• No cross-generation reprocessing of duplicate groups — dedupe phase ~294 s → ~188 s (~36%), kernel dedupe traffic roughly halved.
• Batched SQLite transactions (~10 s cadence) — ~24% faster rescans, file-lock syscalls cut from hundreds of thousands to a few hundred.
• Parallel directory walk and a compact 64-bit path-hash index for large trees.

🎯 Reporting & correctness

• Honest reclaimed-space reporting — the summary now shows the disk space actually freed (one physical copy kept per group), not the fiemap "net change in shared extents" that double-counted pairs. The stable net change in shared extents line is still emitted for scripts (piped / -q).
• Safe concurrent report modes — --stats/--history/--json open the hashfile read-only, so they're safe to run while a scan or dedupe is in progress.
• Hardlink safety fix — INSERT OR REPLACE could cascade-delete other hardlinks' rows and silently empty the hashfile; guarded, with a regression test.
• Permission hint when a hashfile can't be opened; several dedupe-progress display fixes.

🧹 Housekeeping

• Removed --fdupes and legacy/testing options (--hash-threads, -D, --read/--write-hashes); project-wide dead-code cleanup.
• Reworked README leading with the value proposition; build-dependency install commands for Fedora and Debian/Ubuntu.

📦 Install

make && sudo make install          # oans + a duperemove compat symlink
sudo make install-systemd          # optional: the oans@ timer/service templates

Requires Linux 3.13+ on btrfs or XFS. Build deps and full details in the README.

oans v1.1.1

A small follow-up to 1.1.0.

Fixed / improved

• Rebuilt hashfiles are now auto-compacted. A hashfile built from scratch — a brand-new one, or one recreated after a format/version change (such as the 1.1.0 upgrade) — is written at SQLite "insert density" and ends up ~15–20% larger than necessary. oans now runs a one-off VACUUM at the end of such a run (printing Compacting the rebuilt hashfile ...), so it lands compact automatically. If you upgraded to 1.1.0 and ran a manual VACUUM to shrink your hashfile, that's no longer needed. Normal incremental runs are unchanged.

Everything else carries over from 1.1.0.

oans v1.1.0

oans 1.1.0 builds on 1.0.0 with automatic hashfile maintenance, a correctness fix, and a distinct on-disk identity.

⚠️ One-time hashfile rebuild on upgrade

The hashfile format moves to 5.0 and is now branded with an oans application_id. The first oans run after upgrading will print Recreating hashfile .. and do one full re-scan to rebuild it. This is expected and only happens once; the hashfile is just a cache.

Highlights

• Automatic deleted-file pruning. Files removed from disk are now dropped from the hashfile automatically on the next scan (their rows and cascaded hashes), so a long-lived hashfile no longer grows without bound and the dedupe phase stops loading phantom groups for gone files. Freed space is reclaimed (VACUUM) once enough of the file is unused. It's existence-based (stat), so scanning a subdirectory or sharing one hashfile across trees never prunes files that still exist. A "seen-set" built during the scan means a no-op rescan does zero extra stat()s — no steady-state cost.
• Distinct hashfile identity. Every hashfile is stamped with a SQLite application_id ("oans") and the format is bumped to 5.0 (a clean break from duperemove's 4.x line). oans strictly refuses any hashfile that isn't branded as its own, so oans and duperemove can never mis-read each other's files.
• Correctness fix. Fixed an uninitialised-memory read when loading the hashfile config UUID (uuid_parse on an unterminated buffer). Added a valgrind suppressions file so the whole scan/dedupe/prune path is verifiably clean.

Notes

Benchmarks and behavior otherwise carry over from 1.0.0. On compressed filesystems, judge reclaimed space with compsize Disk Usage, not the logical "Deduplicated" figure. Original duperemove by Mark Fasheh and contributors; licensed GPLv2.

oans v1.0.0

oans is a friendly, faster fork of duperemove for btrfs & xfs deduplication. All of the original design and code is the work of Mark Fasheh and the upstream contributors; this is the first tagged release of the fork, breaking from the inherited 0.15.x line so versions no longer clash with upstream.

Drop-in: the oans binary installs a duperemove compatibility symlink, and the hashfile format and CLI are unchanged from upstream.

Highlights

• Skip already-shared files — files whose extents are already shared are detected up front and skipped. On an already-deduped tree (~2M files, ~230 GiB on btrfs): ~90s vs ~11m for upstream 0.15.2.
• No cross-generation reprocessing — large duplicate groups spanning many dedupe passes are handled once instead of re-checked each pass: ~36% faster on a real 2M-file tree, with accurate dedup accounting.
• Batched hashfile transactions — collapses a per-file SQLite lock storm (hundreds of thousands of F_SETLK → a few hundred); ~24% faster rescans.
• Parallel directory walk (--io-threads), with a default tuned to where btrfs metadata contention plateaus.
• Compact path-hash index for faster path lookups on large trees.
• Hardlink safety fix — guards an INSERT OR REPLACE cascade that could silently empty the hashfile (with a regression test).
• Clearer output — human-readable colorful summary and a live dedupe progress bar with throughput/ETA.
• Tests & CI — a Python integration suite (57 tests) driving the built binary against a scratch tree, plus GitHub Actions.

Benchmarks are real but workload-dependent (the already-deduped case is the skip's best case). On compressed filesystems, judge reclaimed space with compsize Disk Usage, not the logical "Deduplicated" figure.

Install

make && sudo make install    # installs oans + a duperemove compat symlink

Requires glib2, sqlite3, libxxhash ≥ 0.8, util-linux, libbsd; Linux kernel ≥ 3.13.

Credits

Original author: Mark Fasheh and the upstream duperemove contributors — https://github.com/markfasheh/duperemove. Licensed GPLv2.