Intra-file chunked hashing (#88)

[martinus]

Closes #88. Splits a large file into chunks hashed in parallel across the IO threads, so a scan dominated by one very large file uses more than one thread. Still draft — the notable item for your call is the default (see below).

Design

A per-byte whole-file checksum can't be split across chunks (XXH128 isn't additive), so the file digest is redefined first, then the chunking is built on top.

1. File digest = fold of the block (loff, digest) array (commit 1). Layout-independent (a block is a fixed 128 KiB file-offset window, not a physical extent → byte-identical files match regardless of fragmentation) and chunk-independent (one pass or N chunks fold the same sequence). Streamed O(1) for whole-file scans; block records are retained only for block-dedup or a chunk merge. Subsumes the More efficient handling of large very sparse files #87 hole handling. DB_FILE_MINOR 5.0→5.1 (one-time rebuild); block/extent digests unchanged.
2. Chunked scanning + merge barrier (commit 2). A chunk is just the existing scan restricted to [start, end); boundaries sit on blocksize-aligned extent ends so blocks and extents stay whole and mapped-byte budgeting composes with More efficient handling of large very sparse files #87. A refcounted scan_job collects per-chunk hashes; the last chunk to finish (atomic dec-and-test) folds the digest and stores everything in one transaction. The K=1 path is unchanged.
3. --chunksize + device-aware default (commit 3). Auto-enables with a 1 GiB target on SSD/NVMe/multi-device pools (where concurrent reads help) and stays off on a single spinning disk; --chunksize=0 forces off. Pure perf knob — not persisted in the hashfile.
4. Tests (commit 4): unit tests for the planner (alignment, budget, cap, hole composition, decline cases) and storage_benefits_from_concurrency; integration tests asserting a chunked scan is byte-for-byte identical to an unchunked one (file digest, extents, block hashes), plus no-op rescan and dedupe, gated @requires_btrfs with a fragmentation assertion so they can't pass vacuously.

Correctness

Chunking never changes any digest, block or extent — verified end-to-end: whole vs chunked scans produce identical files.digest, extents and block hashes; cross-dedupe (chunked file vs whole copy) works; digest is stable across runs; valgrind clean on the merge/refcount path.

Spike (validates the premise)

Cold buffered read+XXH128, one large file, single NVMe, fresh slices (O_DIRECT cross-checked): 1 thread 1.85 GiB/s → 4 threads 3.88 GiB/s (~2.1×), saturating ~4 streams. A single synchronous reader gets ~half the device; more on multi-device pools.

For review — the one decision

This bundles a default-behavior change: chunking is now auto-ON for SSD/NVMe/pools (files ≥2 GiB). It's the intended #88 behavior and heavily verified, but it moves a brand-new concurrency path onto the default. Easy to make opt-in instead (resolve the auto default to 0). Happy to split the default-flip into its own commit or hold it if you'd prefer soak time as opt-in first.

Verification

scripts/verify.sh green: build (warnings=errors), 9 unit tests (5078 assertions) + 93 integration tests, valgrind scan/dedupe/replay smoke clean.

🤖 Generated with Claude Code

[martinus]

Parking this rather than merging — after building it out I don't think the cost/benefit lands. Writing up the reasoning here for the record.

What it costs

• Every existing hashfile gets rebuilt. Redefining the file digest as a fold of the block array is what makes chunking possible, but it bumps DB_FILE_MINOR 5.0→5.1, so on upgrade every user eats a one-time full re-scan — a cost paid by everyone, not just people with giant files.
• A whole new concurrency path to own. Refcounted scan_job, an atomic dec-and-test merge barrier, and a chunk planner (alignment / mapped-byte budget / hole composition). ~964/249 lines of surface that has to stay correct forever.
• Narrow payoff. The win only shows up for a single file ≥2 GiB on SSD/NVMe/multi-device — the spike is ~2.1× on one large file on NVMe. For the common many-files workload the existing per-file parallelism already saturates the pool, so chunking does nothing there.

Why it's mostly the wrong tool for #88
The symptom in #88 is threads going idle because one huge file gets hashed last while everything else is already done. For any workload with more than one big file that's a scheduling problem, not a "split the file" problem. Longest-Processing-Time-first — hand the biggest queued file to the next free thread — keeps the tail packed at essentially zero cost: no digest change, no schema bump, no new concurrency path, just an ordering on the scan pool we already have. I'm putting that on a separate branch.

The one case LPT doesn't cover is a single file larger than the sum of everything else (a lone 50 GiB file among small ones) — only chunking parallelizes that. It's rare, and if it ever becomes a real need we can add chunking behind an opt-in flag without the digest redefinition that forces the global rebuild.

So: holding this in favor of the LPT branch. Keeping it open as a draft for reference in case the single-giant-file case ever justifies revisiting.

[martinus]

Closing in favor of #91 (largest-file-first scheduling), which merged to master as the cheaper solution to #88 — no digest redefinition, no hashfile rebuild, no new concurrency path. Leaving the branch intact for reference if the single-giant-file case (the one thing LPT can't parallelize) ever justifies revisiting intra-file chunking.