Skip to content

oans v1.4.0

Choose a tag to compare

View all tags
@martinus martinus released this 24 Jul 12:17
v1.4.0
0a52d85
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

oans 1.4.0 is a large feature release focused on the dedupe phase, a
unified live progress UI, and honest observability — all backward
compatible with 1.3.0 (same CLI superset, same hashfile schema, hashfiles
carry over).

Highlights

  • Fast dedupe on trees larger than RAM (~13× vs duperemove 0.15.2). When the
    working set doesn't fit in the page cache, the kernel's FIDEDUPERANGE byte
    compare re-reads cold. oans now keeps just-hashed data warm and prefetches each
    dedupe round as fast sequential I/O (#107). Median hash+dedupe on a
    cache-capped ~10.5 GiB tree: 13.8 s vs 179.7 s, at ~2× lower RSS and a
    ~1.8× smaller hashfile — byte-for-byte identical sharing. See
    docs/benchmarks.md.
  • Streaming dedupe pipeline. The dedupe phase runs one persistent worker pool
    with a bounded double-buffered producer, so peak RSS stays low and flat
    regardless of how many duplicate groups are in flight (#111, #112, #116).
  • Unified live progress UI across scanning / hashing / dedupe / done, with a
    byte-weighted smooth dedupe bar and a weighted-progress scan ETA.

Dedupe

  • Byte-weighted, smooth dedupe progress bar — moves 0→100% by kernel byte-verify
    volume, even through one giant group (#111).
  • Order-independent extent loader; whole-file dup members excluded via a
    correlated probe (#112).
  • Streaming pipeline with a persistent pool and generation-ordered watermark,
    preserving the Ctrl+C safety invariant (#116).
  • Keep hashed data in the page cache for the dedupe phase and prefetch each
    FIDEDUPERANGE round (#107).

Scan & progress

  • Weighted-progress scan ETA (#98); retuned HDD per-file weight (#101);
    "mapping:" phase shown during pre-read setup (#99).
  • Hash largest files first (LPT scheduling) to shrink the idle tail (#91).
  • Skip all-hole blocks in sparse files (#87, #89).
  • Refuse unsupported-fs roots up front instead of failing silently (#97); XFS
    identified without root, with a loud failure when it can't (#85).
  • Warn instead of silently skipping paths over PATH_MAX (#115).
  • Grow hash arrays geometrically (#95); block-aligned block-count estimates (#94).
  • Scan diagnostics: contention/starvation counters, honest small-file ETA, no
    more idle-flicker (#93).

Observability & reporting

  • --progress=json: machine-readable per-phase progress for dashboards.
  • --stats relabels "reclaimable" as "duplicated" (a logical figure, not a
    to-do) (#84).
  • Self-contained --help/usage instead of shelling out to man (#86).

Reliability & CI

  • Fix a use-after-free and leak when recreating a rejected hashfile (#105).
  • CI runs the unit + integration suites under valgrind, on btrfs and XFS (#106).

Docs & tooling

  • Consolidated all benchmark information into a single
    docs/benchmarks.md
    with refreshed larger-than-RAM numbers (#109, #118).
  • Consolidated benchmarking into scripts/bench.py (#102).
  • README restructure of "What the fork changes" (#110); note CI covers XFS (#96);
    Fedora 41 libatomic dependency (#82).
Assets 4
Loading