Skip to content

oans v1.6.0

Latest

Choose a tag to compare

@martinus martinus released this 26 Jul 13:47
f390c01

Fixes a bug that silently excluded sparse files from deduplication, and changes
--exclude to .gitignore pattern syntax.

Fixes

  • Sparse files whose size is not a multiple of the block size were silently
    skipped, on every run
    (#152). They were reported as having changed while
    being hashed — which was not true — and then dropped, so they never reached
    the hashfile and were never deduplicated. The tail of a trailing hole was
    rounded down to a block boundary, leaving a final partial block that nothing
    could read; the scan then concluded the file had changed under it and threw
    the digest away. Nothing was stored, so the next run repeated it, forever.
    This hit exactly what sparseness is used for: VM images, database files and
    preallocated media. If you have such files, this release will hash and
    deduplicate them for the first time.
  • Three different conditions all reported the word "changed", with three
    different meanings and no numbers (#152). Each now states what was observed,
    the likely cause, and what follows from it — the scan-phase one in particular
    is routine on files that are still being written and now says so.
  • The dedupe summary no longer reports the wrong cause. One counter was
    incremented both by our own pre-flight size check and by the kernel's
    byte-compare verdict, then printed as a single "changed since scan" figure —
    so every kernel mismatch was attributed to a file change that had not
    happened. The two are now counted and named apart.

Changes

  • --exclude now uses .gitignore pattern syntax (#154, closes #147), the
    same syntax git, ripgrep and fd use:

    Pattern Meaning
    @eaDir, *.iso no / — matches the name at any depth
    /srv/media/cache* leading /absolute, anchored
    Steam/temp interior / — matches at any depth
    cache/ trailing /directories only
    * ? [a-z] ** * stops at /, ** crosses

    Previously patterns were matched with fnmatch() against the whole path, and
    a pattern without a leading / was resolved against the current directory —
    so --exclude node_modules matched at most one literal directory, and usually
    nothing at all, without saying so. The obvious things to write (@eaDir,
    .snapshots, node_modules) were all silent no-ops.

    This is a breaking change. Patterns already stored in a hashfile are
    replayed under the new rules, so a scheduled job may now exclude more, or
    less, than it did. Run your job once by hand and check the output before
    relying on the next timer firing.
    Two things make that easier: a pattern
    matching nothing is now reported as a warning (including under --quiet, so
    it reaches the journal), and a malformed pattern is an error before the scan
    starts rather than something the run ignores while scanning a wider tree.

    Negation (!) is not supported.

  • --autotune has been removed (#153, closes #150). It measured hashing
    throughput at several thread counts and stored the winner in the hashfile.
    Without root it cannot drop the page cache, so it measured memory bandwidth
    instead of disk — which keeps scaling with threads well past the point where
    btrfs metadata contention caps real throughput. It printed a warning and
    persisted the answer anyway, into the hashfile a systemd timer then replays
    forever. On a 32-core NVMe machine it recommended and stored 16 threads where
    the measured plateau, and the built-in default, is 8. --io-threads is still
    sized automatically from the detected storage, and can still be set by hand.

Performance

  • Exclude matching is now independent of pattern count. Patterns compile to
    a single combined PCRE2 regex with JIT enabled, rather than one fnmatch()
    per pattern per file. Measured on a 250k-file tree: ~160 ns more per file in
    the worst case (warm cache, a pattern active), around 0.5% of wall time on the
    most walk-bound workload. GLib was already a dependency, so this adds none.

Documentation

  • The README is up to date with 1.4.0 and 1.5.0, and a stale hashfile-size
    figure that appeared in no benchmark has been corrected (#151).
  • The NAS quick-start lost its --autotune step, and gained the upgrade note
    for the --exclude change.

Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.