Skip to content

bench: add a 'fragmented' profile, the only one with real extent counts - #136

Merged
martinus merged 1 commit into
masterfrom
bench/fragmented-profile
Jul 25, 2026
Merged

bench: add a 'fragmented' profile, the only one with real extent counts#136
martinus merged 1 commit into
masterfrom
bench/fragmented-profile

Conversation

@martinus

Copy link
Copy Markdown
Owner

Why

Every existing profile generates ~1 extent per file — and so does a real source tree. Measured on the actual bench-dedupe.py source:

~/git/linux, 3000-file sample:  mean 1.17 extents/file, median 1, p99 2, max 10

At E=1, an O(E²) per-file scan and an O(E) one are the same work. So per-file extent-list cost was invisible across the whole harness: the get_extent() scan fixed in #134 was 49% of all CPU on a fragmented file, and nothing here could see it — nor would anything catch a reintroduction.

This also answers "should we re-run the larger-than-RAM benchmark after #134?": no. Its source tree is 1.17 extents/file, so that fix cannot move it, and docs/benchmarks.md stays accurate as published. The gap is a missing workload, not a stale number.

What

fragmented: 8 x 512 MiB files, each shredded into ~32k extents by rewriting one block per 32 KiB stride in random order (btrfs CoW makes each rewrite its own extent; ascending order lets the allocator merge them straight back).

That's the shape a long-lived dedupe target degrades into — dedupe fragments what it shares — and it's closer to the NAS/backup workload oans is positioned for than a kernel checkout is.

Adds one Profile field (frag_stride_kb) and a key to PROFILES, per the harness's "add a key, nothing else changes" contract. No new bench-* script.

Two ordering constraints, both found by getting them wrong

  • sync before the rewrites. Rewriting still-dirty pages replaces them in place, no CoW happens, and writeback lays the file out contiguously. My first version produced ~6k extents instead of ~32k and measured nothing at all — base 1.50 vs fix 1.48 wall median, a clean null result from a broken fixture. Same trap as the missing sync between rm and cp in bench-dedupe.py.
  • Fragment after the cp pass, never before. cp writes its destination sequentially, so copying a fragmented original yields a contiguous copy.

_mean_extents() prints the achieved count at build time, so a silently-unfragmented tree announces itself instead of quietly measuring nothing.

Verification

A/B against the pre-#134 binary, 7 warm rounds:

io-threads wall (base → fix) user (base → fix)
8 2.51 → 2.50 (nothing) 3.05 → 1.05 (2.9x)
2 3.32 → 2.51 (1.3x) 2.75 → 0.96 (2.9x)

user always separates. For wall to separate, pair with --io-threads 2 — with 8 files over 8 threads it's one file per thread, so the extra CPU just overlaps. That's the same pairing convention bigfile already documents, and it's in the profile note.

Tree generation is reproducible (32,768 extents/file on a clean regenerate) and the manifest cache correctly skips regeneration on re-runs. scripts/verify.sh passes.

Every existing profile generates ~1 extent per file, and so does a real
source tree - ~/git/linux measures 1.17 mean, p99 2, max 10. Per-file
extent-list cost is therefore invisible across the whole harness: the
O(extents^2) get_extent() scan fixed in #134 was 49% of all CPU on a
fragmented file and no profile here could see it, nor would any of them
catch a reintroduction.

'fragmented' generates 8 x 512 MiB files and shreds each into ~32k extents
by rewriting one block per 32 KiB stride in random order (btrfs CoW makes
each rewrite its own extent; ascending order lets the allocator merge them
back). That is the shape a long-lived dedupe target degrades into, since
dedupe fragments what it shares - and it is closer to the NAS/backup
workload oans targets than a kernel checkout is.

Two ordering constraints, both learned by getting them wrong first:

  - sync before the rewrites. Rewriting still-dirty pages replaces them in
    place, no CoW happens, and writeback lays the file out contiguously:
    the first version produced ~6k extents instead of ~32k and measured
    nothing (base 1.50 vs fix 1.48 wall median, i.e. no signal).
  - fragment after the cp pass, never before: cp writes its destination
    sequentially, so copying a fragmented original yields a contiguous copy.

Verified against the #134 A/B (7 warm rounds, base = the pre-fix binary).
`user` always separates: 3.05s vs 1.05s. For wall to separate too, pair
with --io-threads 2, since 8 files over 8 threads is one file per thread
and the extra CPU merely overlaps:

  --io-threads 8   wall 2.51 vs 2.50   user 3.05 vs 1.05
  --io-threads 2   wall 3.32 vs 2.51   user 2.75 vs 0.96

_mean_extents() prints the achieved count when the tree is built, so a
silently-unfragmented tree announces itself rather than quietly measuring
nothing.

Co-Authored-By: Claude <noreply@anthropic.com>
@martinus
martinus merged commit de2760d into master Jul 25, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant