Skip to content

docs(design): internal xxHash64 file-dedup heuristic#3824

Merged
oferchen merged 1 commit into
masterfrom
docs/xxhash64-dedup-2102
May 7, 2026
Merged

docs(design): internal xxHash64 file-dedup heuristic#3824
oferchen merged 1 commit into
masterfrom
docs/xxhash64-dedup-2102

Conversation

@oferchen
Copy link
Copy Markdown
Owner

@oferchen oferchen commented May 7, 2026

Summary

  • Adds docs/design/xxhash64-file-dedup.md: internal-only heuristic that pre-screens identical files before signature/delta computation.
  • 64-bit xxHash64 fingerprint over (size, mtime, first 4 KB, last 4 KB) computed once at flist build time, then probed against same-flist siblings.
  • Strict three-way gate (fp + size + mtime) with conservative false-negative handling: any mtime divergence falls through to the existing pipeline. Wire output is byte-identical with dedup on or off.

Test plan

  • Doc renders correctly on GitHub (tables, fenced blocks).
  • Wire-compat invariants (Section 8) line up with feedback_no_wire_protocol_features.md and the parent zsync design notes.
  • Implementation / property-test / bench scaffolding PRs (follow-ups under Enforce single branding binary in workspace metadata #2102) reference back to this note.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 7, 2026
@oferchen oferchen merged commit 7e78c81 into master May 7, 2026
8 checks passed
@oferchen oferchen deleted the docs/xxhash64-dedup-2102 branch May 8, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant