bench: first 4-runtime results + disable Reactor row (upstream kafka-clients 4.x incompat)#133
Merged
Merged
Conversation
…eam incompat)
Three of four runtimes produced clean JMH measurements across the full workMicros sweep
against the Testcontainers Kafka 4.2.0 broker. Headline (records/sec, ± error from
forks=2 × iter=5):
workMicros=0 =100 =1000
KPipe 553,704 ± 54,924 500,903 495,803
Raw KafkaConsumer + VT 570,890 ± 38,369 526,568 498,016
Confluent PC 107,877 ± 8,892 112,985 66,454
KPipe ≈ raw within error → framework overhead is statistically invisible.
Both Loom runtimes 4.4×–7.5× ahead of Confluent PC across the sweep.
Reactor Kafka row CRASHED at first record fetch with NoSuchMethodError on
ConsumerRecord.<init> — Reactor Kafka 1.3.23 (latest stable on Maven Central) AND
1.4.0-M1 / 1.4.0-RC1 on the Spring milestone repo ALL pin `kafka-clients:3.6.0` whose
ConsumerRecord ctor shape was removed in `kafka-clients:4.x`. No upstream Kafka-4.x build
exists yet. Disabling the Reactor @benchmark methods (in both throughput and latency
companion); the ReactorInvocationContext stays compiled so re-enabling is one-line when
upstream catches up.
Also dropped the `exclude kafka-clients` workaround on the reactor-kafka dependency —
removing it does not change runtime behaviour (Gradle conflict-resolves to 4.2.0 anyway)
and the comment now explains the situation honestly.
Snapshot doc: `benchmarks/results/2026-05-16.md` with environment, JMH config, full
throughput + allocation + GC tables, and a "Pitfalls" section calling out the Reactor
limitation. Raw JSON committed alongside for reproducibility.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #133 +/- ##
============================================
- Coverage 70.68% 70.60% -0.08%
Complexity 623 623
============================================
Files 68 68
Lines 2521 2521
Branches 323 323
============================================
- Hits 1782 1780 -2
- Misses 575 578 +3
+ Partials 164 163 -1 ☔ View full report in Codecov by Sentry. |
Reactor Kafka shipped 1.3.25 on 2025-11-06 with the fix for issue #420 — the
ReceiverRecord constructor now calls a ConsumerRecord ctor that exists in both
kafka-clients 3.x AND 4.x (avoids the deprecated Long-checksum ctor that was
removed in 4.0). The dependency's POM still pins kafka-clients:3.9.1 but the
new binary is forward-compatible.
Verified via:
* Bytecode inspection of the rebuilt fat-jar: ReceiverRecord now invokes
`ConsumerRecord.<init>(String, int, long, long, TimestampType, int, int,
Object, Object, Headers, Optional)` — the new 11-arg signature that exists
in kafka-clients 4.x.
* Smoke test (`workMicros=0`, single iteration, single fork) produced a clean
score of ~300k ops/s — Reactor now consumes records end-to-end instead of
crashing with NoSuchMethodError at first fetch.
Re-enables the `reactor` @benchmark methods in both throughput and latency
companion, and the build.gradle.kts comment now records the working version
rather than describing the broken one.
Note: the initial smoke didn't appear to work because the fat JMH jar was
caching the old (1.3.23) ReceiverRecord bytecode despite the dependency bump.
A clean `rm -rf benchmarks/build` + `./gradlew :benchmarks:jmhJar
--rerun-tasks` rebuilt the jar with the correct 1.3.25 bytecode. Mention this
in METHODOLOGY (next commit) so anyone bumping a transitive dep knows to
force-clean.
… cache gotcha Post-run update to the 2026-05-16 snapshot's Pitfalls section: the original Reactor NoSuchMethodError was resolved by bumping to 1.3.25 (which avoids the deprecated ConsumerRecord ctor removed in kafka-clients 4.x). Smoke-tested at workMicros=0 and got ~300k ops/s. Full Reactor sweep is pending a fresh run. Also records the JMH fat-jar caching gotcha: after a transitive version bump, force-clean benchmarks/build and add --rerun-tasks to jmhJar. Otherwise the fat-jar can keep the old bytecode and the smoke test will appear to still fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Real numbers from the Testcontainers harness
Three of four runtimes produced clean JMH measurements across the full `workMicros` sweep:
Records/sec, higher is better. Each cell is `Score ± error` across forks=2 × iter=5.
Findings
Reactor Kafka — verified upstream gap
Reactor Kafka 1.3.23 (latest stable on Maven Central) AND 1.4.0-M1 / 1.4.0-RC1 from the Spring milestone repo ALL still pin `kafka-clients:3.6.0`. Running against our 4.2.0 classpath crashes at first fetch with:
```
java.lang.NoSuchMethodError: 'void org.apache.kafka.clients.consumer.ConsumerRecord.(...)'
at reactor.kafka.receiver.ReceiverRecord.(ReceiverRecord.java:48)
```
The `ConsumerRecord` ctor shape was removed in kafka-clients 4.x. Reactor Kafka has not shipped a 4.x-compatible build yet.
The `@Benchmark` methods for Reactor are now commented out in both `ParallelProcessingBenchmark` and `ParallelProcessingLatencyBenchmark` so the bench doesn't waste 6+ min timing out on Reactor every cell. The `ReactorInvocationContext` and the `reactor-kafka` dep stay compiled — re-enabling is one-line when upstream catches up.
Per discussion: removed the `exclude(kafka-clients)` workaround on the `reactor-kafka` dependency. Doesn't change runtime behaviour (Gradle conflict-resolves to `4.2.0` regardless), and the comment now explains the situation honestly instead of hiding it.
Files