oans v1.5.1
A patch release for three problems that shipped in v1.5.0: a hang, a
quadratic scan on fragmented files, and a progress display that could read
like a hang of its own.
Fixes
- Fixed a hang at the end of the scan phase (#138). The scan-phase
progress thread decided when to stop by comparing running sums against
totals, so any file counted into a total but never credited back left the
two unable to ever meet — the thread spun inusleepforever and
pscan_join()blocked for good, with the scan finished and every worker
already exited. The scan phase now takes the same producer-driven shutdown
the dedupe phase already had: only the producer knows when the work is
over, so only the producer ends the thread. It takes a small machine to
hit — 12 copies of the progress tests pinned to 2 cores hang ~15% of the
time, the same load unpinned on 32 cores never does — which is why it
first read as CI flakiness. - Hashing workers no longer sit on a stale "commit" line when the walk is
the bottleneck (#143). A csum worker keeps its progress slot for its
whole life and rolls it from file to file, so between files the line kept
whatever the last file left behind. Invisible on a hashing-bound run; on a
big NAS tree where only a handful of files need hashing, all four workers
showedcommit <last file>for the rest of the run, reading like a hang in
the commit path. A slot now parks as idle once its worker has waited 250 ms
for work — long enough that the sub-millisecond gaps between small files
never flicker.
Performance
- Fixed a quadratic extent scan on fragmented files (#134).
is_area_ignored()calledget_extent()with a NULL resume cursor from
inside its own loop, so every iteration rescanned the extent array from
index 0 — the cursor exists precisely to avoid this, but the hottest caller
never passed one. It is reached once per 1 MiB on every scan, and per block
under--dedupe-options=partial, so the cost scales with extents per file.
On a 1 GiB file with 65536 extents,get_extentwas 48.93% of all CPU;
threading the caller's cursor through drops it off the profile entirely and
halves the wall clock (median 1.76s → 0.88s). Ordinary source trees have
few extents per file and are unchanged — but dedupe itself fragments, so
this hit exactly the re-run workload oans targets. Output is byte-identical.
CI & tooling
- The integration suite runs in parallel (#139): 26.4s → 7.9s on 4 cores,
and the sanitizer legs that re-run all of it at 4–8× the cost benefit most. - Every CI job is bounded by a timeout (#135), after a wedged TSAN leg read
as "in progress" for 23 minutes;tests/run.pyalso warns when run without
ThreadSanitizer suppressions, which otherwise fails ~85 tests as pure artefact. - apt packages are cached between runs (#141), most of the wall time on the
short legs and ~14% of the valgrind leg that sets CI's critical path. - The TSAN pool-lifetime annotation keys on a token, not the pool pointer
(#140) — the pointer is exactly what teardown clears, so reading it to
publish on was itself a race. - Three silent tooling failures are now loud (#137), and the benchmark
harness gained afragmentedprofile (#136) — the only one with real
extent counts, so a reintroduction of #134 would be caught. - A hardlink test no longer asserts an exact extent count (#142); how many
extents a file gets is btrfs's business.
Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.