Skip to content

bench: add Share Consumer (KIP-932) + KPipe KEY_ORDERED arms#138

Merged
eschizoid merged 3 commits into
mainfrom
feature/share-consumer-benchmark
May 29, 2026
Merged

bench: add Share Consumer (KIP-932) + KPipe KEY_ORDERED arms#138
eschizoid merged 3 commits into
mainfrom
feature/share-consumer-benchmark

Conversation

@eschizoid

@eschizoid eschizoid commented May 29, 2026

Copy link
Copy Markdown
Owner

Summary

Adds two runtimes to the competitive ParallelProcessingBenchmark, building on the now-merged key-ordered work (#137):

  • Kafka Share Consumer (KIP-932) — one KafkaShareConsumer; each polled batch is fanned to virtual threads with a per-batch barrier before the next poll (implicit ACCEPT = at-least-once). The controlled apples-to-apples rival of KPipe PARALLEL: both unordered, one process, VT fan-out — only the fetch+acknowledge protocol differs.
  • KPipe KEY_ORDERED — same single consumer, per-key serial queues. The only arm with an ordering guarantee; present to quantify the per-key ordering tax, NOT as a head-to-head (no other runtime, share included, can order).

Both reuse the existing Testcontainers harness, so every variable except the consumption mechanism is pinned: same broker, seed, workMicros sweep, and @OperationsPerInvocation throughput metric.

Supporting changes:

  • Keyed seeding (KEY_CARDINALITY=1000) so KEY_ORDERED is meaningful; the unordered arms ignore keys.
  • Broker env for share groups (share rebalance protocol + __share_group_state topic RF) and a per-group share.auto.offset.reset=earliest set via Admin, so the fresh share group reads the pre-seeded records.
  • METHODOLOGY.md: the apples-to-apples framing, the ordering-tax caveat, and the single-process (not scale-out) scope.

Published results

Full publishing run committed in this PR: benchmarks/results/2026-05-29.md + raw results-2026-05-29.json. fork=2, warmup=3, iter=5, gc profiler. Intel i7-8850H (6c/12t), JDK 25.0.2, Testcontainers Kafka 4.2.0.

Throughput (records/s; see the results file for ± error bars):

Runtime Ordering workMicros=0 workMicros=100 workMicros=1000
Raw KafkaConsumer + VT none 630,515 573,101 573,443
KPipe (PARALLEL) none 525,051 581,500 503,612
KPipe (KEY_ORDERED) per-key 311,854 240,005 89,745
Reactor Kafka none 309,724 78,953 9,011
Confluent Parallel Consumer none 111,541 108,518 64,539
Kafka Share Consumer (932) none 4,839 4,813 4,789

Takeaways (full detail in the results file):

  • KPipe PARALLEL tracks the raw-VT ceiling within ~10% and stays flat under blocking work (Loom).
  • KEY_ORDERED's 312k → 90k drop is the ordering tax, not a loss — still ahead of Confluent/Reactor at 1000 µs.
  • Reactor collapses 310k → 9k under blocking work (core-bound scheduler); Confluent is pool-bound at ~64k.
  • Share is protocol-bound and flat at ~4.8k regardless of workMicros. This is a single share member — the bench does not exercise share's scale-out (members > partitions), so read it as a single-member ceiling in early KIP-932, not "share is slow," and not an ordering verdict against KEY_ORDERED (they aren't rivals).

Caveats (also in the results file): Intel x86 not Apple Silicon (don't compare absolute numbers across machines); workMicros=0 is the noisiest cell; macOS + Docker loopback broker means absolute records/s are low — read orderings, not absolutes.

Test plan

  • Both new arms validated end-to-end on Docker (share group forms, assigns all 8 partitions, consumes seeded records).
  • Full publishing run + dated results file committed.
  • Follow-up (separate): multi-member share-group experiment to evaluate share scale-out (its actual value prop).
  • Follow-up (separate): tune share throughput (max.poll.records, ack batching, acquisition-lock timeout).

Extend ParallelProcessingBenchmark with two new runtimes so the
competitive suite can answer "share consumer vs key-ordered dispatch,"
reusing the existing Testcontainers harness so every variable except the
consumption mechanism is pinned (same broker, seed, workMicros sweep, and
@OperationsPerInvocation throughput metric).

- Share Consumer arm: one KafkaShareConsumer, each polled batch fanned to
  virtual threads with a per-batch barrier before the next poll (implicit
  ACCEPT = at-least-once). This is the controlled apples-to-apples rival of
  KPipe PARALLEL — both unordered, one process, VT fan-out; only the
  fetch+acknowledge protocol differs.
- KPipe KEY_ORDERED arm: same single consumer, per-key serial queues. The
  only arm with an ordering guarantee — present to quantify the per-key
  ordering tax, NOT as a head-to-head (no other runtime, share included,
  can order).
- Keyed seeding (KEY_CARDINALITY=1000) so KEY_ORDERED is meaningful;
  unordered arms ignore keys.
- Broker env for share groups (share rebalance protocol + __share_group_state
  topic RF) and a per-group share.auto.offset.reset=earliest via Admin so the
  fresh share group reads pre-seeded records.
- METHODOLOGY.md documents the apples-to-apples framing, the ordering-tax
  caveat, the single-process (not scale-out) scope, and the share-group
  setup as a verify-on-first-Docker-run item.

Compiles against the 4.2.0 client API; NOT run (no Docker in the authoring
environment), so the share-group runtime path is unvalidated — see the
METHODOLOGY caveat.
…ound

A smoke run on Testcontainers Kafka 4.2.0 confirmed the share group forms,
assigns all 8 partitions, and consumes the seeded records (so the earliest
offset reset took). Replace the "unvalidated / not run" caveat with the
validated state, and record that the share arm's throughput is ~flat across
workMicros — protocol/ack-bound, not work-bound — so it needs tuning before
its numbers are quoted.
@codecov

codecov Bot commented May 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@54bf55f). Learn more about missing BASE report.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #138   +/-   ##
=======================================
  Coverage        ?   74.64%           
  Complexity      ?      763           
=======================================
  Files           ?       76           
  Lines           ?     2930           
  Branches        ?      367           
=======================================
  Hits            ?     2187           
  Misses          ?      568           
  Partials        ?      175           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

First full publishing run (fork=2, warmup=3, iter=5, gc profiler) on the
6-arm ParallelProcessingBenchmark. Headlines (Intel i7-8850H, JDK 25,
Testcontainers Kafka 4.2.0; records/sec, see the file for error bars):

- Raw VT ~630k and KPipe PARALLEL ~525–581k stay flat across workMicros —
  Loom absorbs blocking work; KPipe tracks the raw baseline within ~10%.
- KPipe KEY_ORDERED shows the ordering tax: 312k → 90k as work rises, still
  ahead of Confluent/Reactor at 1000us.
- Reactor collapses 310k → 9k under blocking work (core-bound scheduler);
  Confluent is pool-bound at ~64k.
- Share consumer is protocol-bound and flat at ~4.8k regardless of work —
  but this is a SINGLE share member; the bench does not test share's
  scale-out, so it's a single-member ceiling, not a verdict.

Raw JMH JSON committed alongside. Latency + batch-sink sections marked N/A
(only ParallelProcessingBenchmark was run). METHODOLOGY table re-aligned by
spotless.
@eschizoid eschizoid merged commit 1c564cb into main May 29, 2026
4 checks passed
@eschizoid eschizoid deleted the feature/share-consumer-benchmark branch May 30, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant