docs(design): intra-file parallel rolling-hash for large files (#2206)#4222
Merged
Merged
Conversation
Records the design space for splitting a single large basis file into overlapping windows and running parallel rolling-hash matchers. Cites the current sequential scan in crates/matching and crates/transfer, proposes the MIN_PARALLEL_FILE_SIZE_BYTES threshold, and recommends deferring the implementation until MD4/MD5 multibuf SIMD benches (#4189, #4191) show single-thread strong-checksum verify headroom is exhausted.
oferchen
added a commit
that referenced
this pull request
May 18, 2026
#4222) Records the design space for splitting a single large basis file into overlapping windows and running parallel rolling-hash matchers. Cites the current sequential scan in crates/matching and crates/transfer, proposes the MIN_PARALLEL_FILE_SIZE_BYTES threshold, and recommends deferring the implementation until MD4/MD5 multibuf SIMD benches (#4189, #4191) show single-thread strong-checksum verify headroom is exhausted.
oferchen
added a commit
that referenced
this pull request
May 18, 2026
#4222) Records the design space for splitting a single large basis file into overlapping windows and running parallel rolling-hash matchers. Cites the current sequential scan in crates/matching and crates/transfer, proposes the MIN_PARALLEL_FILE_SIZE_BYTES threshold, and recommends deferring the implementation until MD4/MD5 multibuf SIMD benches (#4189, #4191) show single-thread strong-checksum verify headroom is exhausted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/design/delta-intra-file-parallel.mdcapturing the design space for splitting a single large basis file (>= ~100 MB) into overlapping windows and running N parallel rolling-hash matchers, with the merged token stream emitted in input-offset order.crates/matching/src/generator.rs:141 DeltaGenerator::generate, hot loop at:196-263) and its sole production caller (crates/transfer/src/generator/delta.rs:171).MIN_PARALLEL_FILE_SIZE_BYTES = 128 * 1024 * 1024next to existing thresholds incrates/transfer/src/parallel_io.rs, with rationale tied tocalculate_block_lengthatcrates/signature/src/block_size.rs:129.match.creference doc (docs(design): upstream match.c and token.c reference for delta work (#1580) #4202), MD4/MD5 multibuf SIMD benches (perf(checksums): bench AVX2 multi-buffer MD4 scalar vs batch (#2100) #4189, bench(checksums): MD5 scalar vs batch throughput (#2101) #4191), zsync adversarial fixtures (Respect last occurrence for tri-state toggles #2079, Ensure delete-after uses content verification before skipping copy #2080), zsync benches (Ensure identical timestamp files are compared by content #2063, Ensure --ignore-times forces a rewrite #2067, Ensure copy skip heuristics verify file content #2071), and the sequence-indexedReorderBufferwork (Ensure release task uploads cross-compiled binaries #1885).Recommendation
Defer. Do not implement intra-file parallel rolling-hash until the MD4/MD5 multibuf SIMD benches (#4189, #4191) show single-thread strong-checksum verify is no longer the hot stage on representative workloads. Either lifts single-thread throughput enough that intra-file parallelism becomes unnecessary, or it does not - the decision must rest on bench numbers, not design speculation. A prototype-behind-feature path is rejected because behavioural divergence between gated and default builds is exactly what the golden-byte parity tests are designed to catch.
Wire-format
No wire-format change. Token stream MUST be byte-identical to the serial matcher's output for every input.
crates/protocol/tests/golden/and the interop matrix must remain green for any future implementation PR.Test plan
cargo fmt --allclean (verified before push).