bench: add a 'fragmented' profile, the only one with real extent counts - #136
Merged
Conversation
Every existing profile generates ~1 extent per file, and so does a real source tree - ~/git/linux measures 1.17 mean, p99 2, max 10. Per-file extent-list cost is therefore invisible across the whole harness: the O(extents^2) get_extent() scan fixed in #134 was 49% of all CPU on a fragmented file and no profile here could see it, nor would any of them catch a reintroduction. 'fragmented' generates 8 x 512 MiB files and shreds each into ~32k extents by rewriting one block per 32 KiB stride in random order (btrfs CoW makes each rewrite its own extent; ascending order lets the allocator merge them back). That is the shape a long-lived dedupe target degrades into, since dedupe fragments what it shares - and it is closer to the NAS/backup workload oans targets than a kernel checkout is. Two ordering constraints, both learned by getting them wrong first: - sync before the rewrites. Rewriting still-dirty pages replaces them in place, no CoW happens, and writeback lays the file out contiguously: the first version produced ~6k extents instead of ~32k and measured nothing (base 1.50 vs fix 1.48 wall median, i.e. no signal). - fragment after the cp pass, never before: cp writes its destination sequentially, so copying a fragmented original yields a contiguous copy. Verified against the #134 A/B (7 warm rounds, base = the pre-fix binary). `user` always separates: 3.05s vs 1.05s. For wall to separate too, pair with --io-threads 2, since 8 files over 8 threads is one file per thread and the extra CPU merely overlaps: --io-threads 8 wall 2.51 vs 2.50 user 3.05 vs 1.05 --io-threads 2 wall 3.32 vs 2.51 user 2.75 vs 0.96 _mean_extents() prints the achieved count when the tree is built, so a silently-unfragmented tree announces itself rather than quietly measuring nothing. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Every existing profile generates ~1 extent per file — and so does a real source tree. Measured on the actual
bench-dedupe.pysource:At E=1, an O(E²) per-file scan and an O(E) one are the same work. So per-file extent-list cost was invisible across the whole harness: the
get_extent()scan fixed in #134 was 49% of all CPU on a fragmented file, and nothing here could see it — nor would anything catch a reintroduction.This also answers "should we re-run the larger-than-RAM benchmark after #134?": no. Its source tree is 1.17 extents/file, so that fix cannot move it, and
docs/benchmarks.mdstays accurate as published. The gap is a missing workload, not a stale number.What
fragmented: 8 x 512 MiB files, each shredded into ~32k extents by rewriting one block per 32 KiB stride in random order (btrfs CoW makes each rewrite its own extent; ascending order lets the allocator merge them straight back).That's the shape a long-lived dedupe target degrades into — dedupe fragments what it shares — and it's closer to the NAS/backup workload oans is positioned for than a kernel checkout is.
Adds one
Profilefield (frag_stride_kb) and a key toPROFILES, per the harness's "add a key, nothing else changes" contract. No newbench-*script.Two ordering constraints, both found by getting them wrong
syncbefore the rewrites. Rewriting still-dirty pages replaces them in place, no CoW happens, and writeback lays the file out contiguously. My first version produced ~6k extents instead of ~32k and measured nothing at all —base 1.50 vs fix 1.48wall median, a clean null result from a broken fixture. Same trap as the missingsyncbetweenrmandcpinbench-dedupe.py.cppass, never before.cpwrites its destination sequentially, so copying a fragmented original yields a contiguous copy._mean_extents()prints the achieved count at build time, so a silently-unfragmented tree announces itself instead of quietly measuring nothing.Verification
A/B against the pre-#134 binary, 7 warm rounds:
useralways separates. For wall to separate, pair with--io-threads 2— with 8 files over 8 threads it's one file per thread, so the extra CPU just overlaps. That's the same pairing conventionbigfilealready documents, and it's in the profile note.Tree generation is reproducible (32,768 extents/file on a clean regenerate) and the manifest cache correctly skips regeneration on re-runs.
scripts/verify.shpasses.