Skip to content

docs(audits): profile parallel signature computation scaling (#1085)#3850

Merged
oferchen merged 1 commit into
masterfrom
docs/parallel-signature-scaling-1085
May 7, 2026
Merged

docs(audits): profile parallel signature computation scaling (#1085)#3850
oferchen merged 1 commit into
masterfrom
docs/parallel-signature-scaling-1085

Conversation

@oferchen
Copy link
Copy Markdown
Owner

@oferchen oferchen commented May 7, 2026

Summary

Adds docs/audits/parallel-signature-scaling.md covering the signature pipeline under crates/signature/, per-block rolling and strong-checksum costs at varying block sizes, and projected 1/4/16/64 thread scaling on 100 MB / 1 GB / 10 GB basis files.

  • Maps the three generators (sequential batched, pipelined, rayon-parallel) and identifies that the parallel path conflates the SIMD batch boundary with the rayon partition stride.
  • Proposes five improvements: adaptive rayon chunk sizing, per-thread scratch reuse, overlapping the I/O prologue with the existing pipelined reader, SIMD-batched rolling checksum, and GPU offload as deferred future work.

Tracking: oc-rsync task #1085.

Test plan

  • cargo fmt --all -- --check
  • No code changes; doc-only addition under docs/audits/

Document the signature pipeline (sequential, pipelined, rayon-parallel)
under crates/signature/, capture per-block rolling and strong-checksum
costs, project 1/4/16/64 thread scaling on 100 MB / 1 GB / 10 GB bases,
and propose tuned chunk sizing, per-thread scratch reuse, prologue
overlap with the pipelined reader, SIMD-batched rolling checksum, and
GPU offload as deferred future work.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 7, 2026
@oferchen oferchen merged commit 1114673 into master May 7, 2026
8 checks passed
@oferchen oferchen deleted the docs/parallel-signature-scaling-1085 branch May 8, 2026 03:50
oferchen added a commit that referenced this pull request May 18, 2026
…3850)

Document the signature pipeline (sequential, pipelined, rayon-parallel)
under crates/signature/, capture per-block rolling and strong-checksum
costs, project 1/4/16/64 thread scaling on 100 MB / 1 GB / 10 GB bases,
and propose tuned chunk sizing, per-thread scratch reuse, prologue
overlap with the pipelined reader, SIMD-batched rolling checksum, and
GPU offload as deferred future work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant