Skip to content

feat(server): return base_offset and partition_id from send_messages#3101

Closed
sujitn wants to merge 3 commits into
apache:masterfrom
sujitn:feat/send-messages-return-offset
Closed

feat(server): return base_offset and partition_id from send_messages#3101
sujitn wants to merge 3 commits into
apache:masterfrom
sujitn:feat/send-messages-return-offset

Conversation

@sujitn
Copy link
Copy Markdown

@sujitn sujitn commented Apr 12, 2026

The offset assigned during append was computed but discarded. Propagate it through ShardResponse::SendMessages and include it in the binary protocol response.

Wire format change:
Before: [status:4 LE][length:4 LE (0)]
After: [status:4 LE][length:4 LE (12)][base_offset:8 LE][partition_id:4 LE]

existing SDK clients (excluding node) read the length field and skip unknown body bytes. The send_messages wrapper discards the response body and returns Ok(()).

Fixed node SDK to ignore length

Ref: #3044

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

If AI tools were used, please answer:

  1. Which tools?
    Claude

  2. Scope of usage?
    SDK validation

  3. How did you verify the generated code works correctly?

  • 64/64 server unit tests pass (cargo test -p server --lib)
  • 36/36 offset retrieval integration tests pass, these send messages via the SDK
  • 87/87 concurrent produce/consume integration tests pass
  • 6/6 authentication integration tests pass
  • cargo clippy -p server --all-targets -- -D warnings - 0 warnings
  1. Can you explain every line of the code if asked?
    Yes

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.73%. Comparing base (aa34566) to head (c130f35).
⚠️ Report is 23 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3101      +/-   ##
============================================
- Coverage     72.75%   72.73%   -0.02%     
  Complexity      943      943              
============================================
  Files          1117     1117              
  Lines         96343    96335       -8     
  Branches      73544    73552       +8     
============================================
- Hits          70092    70067      -25     
- Misses        23700    23703       +3     
- Partials       2551     2565      +14     
Components Coverage Δ
Rust Core 73.50% <ø> (-0.01%) ⬇️
Java SDK 62.30% <ø> (ø)
C# SDK 69.10% <ø> (-0.31%) ⬇️
Python SDK 81.43% <ø> (ø)
Node SDK 91.53% <100.00%> (ø)
Go SDK 38.97% <ø> (ø)
Files with missing lines Coverage Δ
.../binary/handlers/messages/send_messages_handler.rs 100.00% <ø> (ø)
core/server/src/http/http_shard_wrapper.rs 74.12% <ø> (-0.18%) ⬇️
core/server/src/shard/handlers.rs 77.83% <ø> (-0.12%) ⬇️
core/server/src/shard/system/messages.rs 87.84% <ø> (+0.08%) ⬆️
core/server/src/shard/transmission/frame.rs 100.00% <ø> (ø)
foreign/node/src/client/client.utils.ts 89.89% <100.00%> (ø)

... and 16 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sujitn sujitn force-pushed the feat/send-messages-return-offset branch 2 times, most recently from b3ad595 to ee34994 Compare April 12, 2026 13:26
Comment thread core/server/src/binary/handlers/messages/send_messages_handler.rs Outdated
@sujitn sujitn force-pushed the feat/send-messages-return-offset branch 2 times, most recently from 265d6be to d2b2e3d Compare April 13, 2026 05:51
Comment thread core/server/src/binary/handlers/messages/send_messages_handler.rs Outdated
The offset assigned during append was computed but discarded.
Propagate it through ShardResponse::SendMessages and include
it in the binary protocol response.

Wire format change:
  Before: [status:4 LE][length:4 LE (0)]
  After:  [status:4 LE][length:4 LE (12)][base_offset:8 LE][partition_id:4 LE]

Backwards compatible — existing SDK clients read the length field
and skip unknown body bytes. The send_messages wrapper discards
the response body and returns Ok(()).

Ref: apache#3044
@sujitn sujitn force-pushed the feat/send-messages-return-offset branch from 703510e to a685301 Compare April 13, 2026 18:08
@sujitn sujitn requested a review from hubcio April 13, 2026 21:17
@hubcio
Copy link
Copy Markdown
Contributor

hubcio commented Apr 13, 2026

I'm gonna review this properly once we finalize current release 0.8.0.

@lukaszzborek
Copy link
Copy Markdown
Contributor

I think we should at least create issues for every sdk to implement that change in api (just after merge)

@github-actions
Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added stale Inactive issue or pull request and removed stale Inactive issue or pull request labels Apr 22, 2026
@hubcio
Copy link
Copy Markdown
Contributor

hubcio commented Apr 29, 2026

We decided to postpone this until cluster v0 is merged.
I'll resume discussion at some point in future one of these threads:
#3081
#6

Until then, no code should be written besides spikes/PoCs. Of course, design studies can be continued.

@hubcio hubcio closed this Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants