Skip to content

perf(server-ng): compute batch checksum in a single produce pass - #3746

Merged
hubcio merged 2 commits into
masterfrom
feat/server-ng-perf
Aug 4, 2026
Merged

perf(server-ng): compute batch checksum in a single produce pass#3746
hubcio merged 2 commits into
masterfrom
feat/server-ng-perf

Conversation

@hubcio

@hubcio hubcio commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Produce hashed each batch three times: convert, stamp, and flush
revalidation. The batch checksum is redefined to cover the six header
meta fields plus each message's stored checksum field instead of the
whole blob, binding bodies transitively through the per-message
checksums every validating decode re-verifies. Stamping now hashes N*8
bytes instead of the full blob, and locally originated batches decode
trusted; replicated blobs keep a per-message receive gate on
followers, so transit integrity holds end to end.

Message data written by earlier server-ng builds fails checksum
validation after this change; wipe local_data when upgrading.
Per-message checksums are computed in a single oneshot pass over the
encoded record, pinned by a streaming-vs-oneshot reference test.

This is half of the single-node latency parity work; the consumer-side
sealed-segment read-handle LRU and the numa:auto shard-allocation
default were split out into a separate PR.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 24, 2026
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.91176% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.65%. Comparing base (c51b754) to head (15cebf5).

Files with missing lines Patch % Lines
core/server_common/src/send_messages2.rs 94.08% 16 Missing and 15 partials ⚠️
core/partitions/src/iggy_partition.rs 29.41% 10 Missing and 2 partials ⚠️
core/partitions/src/iggy_partitions.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3746       +/-   ##
=============================================
- Coverage     76.20%   63.65%   -12.55%     
  Complexity      969      969               
=============================================
  Files          1325     1324        -1     
  Lines        163090   152239    -10851     
  Branches     135971   125197    -10774     
=============================================
- Hits         124281    96907    -27374     
- Misses        35123    51523    +16400     
- Partials       3686     3809      +123     
Components Coverage Δ
Rust Core 60.76% <91.91%> (-15.41%) ⬇️
Java SDK 62.71% <ø> (ø)
C# SDK 71.13% <ø> (-1.14%) ⬇️
Python SDK 89.45% <ø> (ø)
PHP SDK 84.52% <ø> (ø)
Node SDK 96.34% <ø> (+0.08%) ⬆️
Go SDK 43.08% <ø> (ø)
Files with missing lines Coverage Δ
core/partitions/src/journal.rs 77.21% <100.00%> (ø)
core/server-ng/src/responses.rs 39.76% <ø> (ø)
core/simulator/src/client.rs 60.39% <ø> (ø)
core/partitions/src/iggy_partitions.rs 70.02% <0.00%> (ø)
core/partitions/src/iggy_partition.rs 64.09% <29.41%> (-0.19%) ⬇️
core/server_common/src/send_messages2.rs 87.15% <94.08%> (+18.18%) ⬆️

... and 257 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio
hubcio force-pushed the feat/server-ng-perf branch from 7bf57b2 to 3a944be Compare July 27, 2026 08:59
Comment thread core/server_common/src/send_messages2.rs
Comment thread core/partitions/src/iggy_index_reader.rs
Comment thread core/partitions/src/poll_plan.rs
@hubcio

hubcio commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

@krishvishal tyvm for review, i think i will split this PR into 2 smaller PRs to make it more bearable for review.

EDIT: done

@hubcio
hubcio force-pushed the feat/server-ng-perf branch from 3a944be to a8832ce Compare August 3, 2026 09:04
@hubcio hubcio changed the title perf(server-ng): reach single-node latency parity with legacy server perf(server-ng): compute batch checksum in a single produce pass Aug 3, 2026
@hubcio
hubcio force-pushed the feat/server-ng-perf branch from a8832ce to e87f1e3 Compare August 3, 2026 09:35
Produce hashed each batch three times: convert, stamp, and flush
revalidation. The batch checksum is redefined to cover the six header
meta fields plus each message's stored checksum field instead of the
whole blob, binding bodies transitively through the per-message
checksums every validating decode re-verifies. Stamping now hashes N*8
bytes instead of the full blob, and locally originated batches decode
trusted; replicated blobs keep a per-message receive gate on
followers, so transit integrity holds end to end. Message data written
by earlier server-ng builds fails checksum validation after this
change; wipe local_data when upgrading. Per-message checksums are
computed in a single oneshot pass over the encoded record, pinned by a
streaming-vs-oneshot reference test.

A frame walk stops at the last decodable record, so bytes past
batch_length no longer alter any checksum. Both ingest boundaries
therefore require the frame to end exactly at batch_length: a
SendMessages request or prepare whose size overshoots is now rejected
rather than carrying the surplus to disk, where the flush would write
it and the segment walk would step over it into junk.
@hubcio
hubcio force-pushed the feat/server-ng-perf branch from e87f1e3 to b691d33 Compare August 3, 2026 09:36
spetz pushed a commit that referenced this pull request Aug 4, 2026
…RU (#3806)

A lone consumer crossing a sealed segment degraded to full-segment
scans: rotation drops the writer and the resident index, so every poll
re-opened the file and scanned from byte 0. Sealed segments now share
a per-partition read-state handle caching the read fd and the sparse
index reloaded from the .index file, capped by an LRU so descriptors
stay bounded.

Shard allocation defaults back to numa:auto now that multi-shard
server-ng is stable; the previous single-shard default collapsed all
work onto one core.

This is half of the single-node latency parity work; the produce-side
single-pass batch checksum change lives in #3746.
@hubcio
hubcio merged commit 60b8789 into master Aug 4, 2026
102 checks passed
@hubcio
hubcio deleted the feat/server-ng-perf branch August 4, 2026 11:15
@github-actions github-actions Bot removed the S-waiting-on-review PR is waiting on a reviewer label Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants