Fixes a bug that silently excluded sparse files from deduplication, and changes
--exclude to .gitignore pattern syntax.
Fixes
- Sparse files whose size is not a multiple of the block size were silently
skipped, on every run (#152). They were reported as having changed while
being hashed — which was not true — and then dropped, so they never reached
the hashfile and were never deduplicated. The tail of a trailing hole was
rounded down to a block boundary, leaving a final partial block that nothing
could read; the scan then concluded the file had changed under it and threw
the digest away. Nothing was stored, so the next run repeated it, forever.
This hit exactly what sparseness is used for: VM images, database files and
preallocated media. If you have such files, this release will hash and
deduplicate them for the first time. - Three different conditions all reported the word "changed", with three
different meanings and no numbers (#152). Each now states what was observed,
the likely cause, and what follows from it — the scan-phase one in particular
is routine on files that are still being written and now says so. - The dedupe summary no longer reports the wrong cause. One counter was
incremented both by our own pre-flight size check and by the kernel's
byte-compare verdict, then printed as a single "changed since scan" figure —
so every kernel mismatch was attributed to a file change that had not
happened. The two are now counted and named apart.
Changes
-
--excludenow uses.gitignorepattern syntax (#154, closes #147), the
same syntax git, ripgrep and fd use:Pattern Meaning @eaDir,*.isono /— matches the name at any depth/srv/media/cache*leading /— absolute, anchoredSteam/tempinterior /— matches at any depthcache/trailing /— directories only*?[a-z]***stops at/,**crossesPreviously patterns were matched with
fnmatch()against the whole path, and
a pattern without a leading/was resolved against the current directory —
so--exclude node_modulesmatched at most one literal directory, and usually
nothing at all, without saying so. The obvious things to write (@eaDir,
.snapshots,node_modules) were all silent no-ops.This is a breaking change. Patterns already stored in a hashfile are
replayed under the new rules, so a scheduled job may now exclude more, or
less, than it did. Run your job once by hand and check the output before
relying on the next timer firing. Two things make that easier: a pattern
matching nothing is now reported as a warning (including under--quiet, so
it reaches the journal), and a malformed pattern is an error before the scan
starts rather than something the run ignores while scanning a wider tree.Negation (
!) is not supported. -
--autotunehas been removed (#153, closes #150). It measured hashing
throughput at several thread counts and stored the winner in the hashfile.
Without root it cannot drop the page cache, so it measured memory bandwidth
instead of disk — which keeps scaling with threads well past the point where
btrfs metadata contention caps real throughput. It printed a warning and
persisted the answer anyway, into the hashfile a systemd timer then replays
forever. On a 32-core NVMe machine it recommended and stored 16 threads where
the measured plateau, and the built-in default, is 8.--io-threadsis still
sized automatically from the detected storage, and can still be set by hand.
Performance
- Exclude matching is now independent of pattern count. Patterns compile to
a single combined PCRE2 regex with JIT enabled, rather than onefnmatch()
per pattern per file. Measured on a 250k-file tree: ~160 ns more per file in
the worst case (warm cache, a pattern active), around 0.5% of wall time on the
most walk-bound workload. GLib was already a dependency, so this adds none.
Documentation
- The README is up to date with 1.4.0 and 1.5.0, and a stale hashfile-size
figure that appeared in no benchmark has been corrected (#151). - The NAS quick-start lost its
--autotunestep, and gained the upgrade note
for the--excludechange.
Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.