Skip to content

oans v1.5.1

Choose a tag to compare

@martinus martinus released this 25 Jul 20:11
114efaf

A patch release for three problems that shipped in v1.5.0: a hang, a
quadratic scan on fragmented files, and a progress display that could read
like a hang of its own.

Fixes

  • Fixed a hang at the end of the scan phase (#138). The scan-phase
    progress thread decided when to stop by comparing running sums against
    totals, so any file counted into a total but never credited back left the
    two unable to ever meet — the thread spun in usleep forever and
    pscan_join() blocked for good, with the scan finished and every worker
    already exited. The scan phase now takes the same producer-driven shutdown
    the dedupe phase already had: only the producer knows when the work is
    over, so only the producer ends the thread. It takes a small machine to
    hit — 12 copies of the progress tests pinned to 2 cores hang ~15% of the
    time, the same load unpinned on 32 cores never does — which is why it
    first read as CI flakiness.
  • Hashing workers no longer sit on a stale "commit" line when the walk is
    the bottleneck
    (#143). A csum worker keeps its progress slot for its
    whole life and rolls it from file to file, so between files the line kept
    whatever the last file left behind. Invisible on a hashing-bound run; on a
    big NAS tree where only a handful of files need hashing, all four workers
    showed commit <last file> for the rest of the run, reading like a hang in
    the commit path. A slot now parks as idle once its worker has waited 250 ms
    for work — long enough that the sub-millisecond gaps between small files
    never flicker.

Performance

  • Fixed a quadratic extent scan on fragmented files (#134).
    is_area_ignored() called get_extent() with a NULL resume cursor from
    inside its own loop, so every iteration rescanned the extent array from
    index 0 — the cursor exists precisely to avoid this, but the hottest caller
    never passed one. It is reached once per 1 MiB on every scan, and per block
    under --dedupe-options=partial, so the cost scales with extents per file.
    On a 1 GiB file with 65536 extents, get_extent was 48.93% of all CPU;
    threading the caller's cursor through drops it off the profile entirely and
    halves the wall clock (median 1.76s → 0.88s). Ordinary source trees have
    few extents per file and are unchanged — but dedupe itself fragments, so
    this hit exactly the re-run workload oans targets. Output is byte-identical.

CI & tooling

  • The integration suite runs in parallel (#139): 26.4s → 7.9s on 4 cores,
    and the sanitizer legs that re-run all of it at 4–8× the cost benefit most.
  • Every CI job is bounded by a timeout (#135), after a wedged TSAN leg read
    as "in progress" for 23 minutes; tests/run.py also warns when run without
    ThreadSanitizer suppressions, which otherwise fails ~85 tests as pure artefact.
  • apt packages are cached between runs (#141), most of the wall time on the
    short legs and ~14% of the valgrind leg that sets CI's critical path.
  • The TSAN pool-lifetime annotation keys on a token, not the pool pointer
    (#140) — the pointer is exactly what teardown clears, so reading it to
    publish on was itself a race.
  • Three silent tooling failures are now loud (#137), and the benchmark
    harness gained a fragmented profile (#136) — the only one with real
    extent counts, so a reintroduction of #134 would be caught.
  • A hardlink test no longer asserts an exact extent count (#142); how many
    extents a file gets is btrfs's business.

Deduplication goes through the kernel's FIDEDUPERANGE ioctl, which
byte-compares every range before sharing it.