fix(bench): give Spring Boot the same tuned low-concurrency run CWIST gets - #33
Merged
Conversation
… gets
The "tuned low-latency run" (wrk -t4 -c100, vs the main -t12 -c400
profile) was CWIST-only: a second, dedicated boot measured under lower
concurrency, then published in the README as a standalone headline
number ("Tuned low-latency run: X req/s at Yms") with no equivalent
Spring Boot figure anywhere nearby. A reader comparing that line
against the Spring Boot row in the main summary table above was
comparing CWIST's best case against a number Spring was never measured
at - a real bias in CWIST's favor, and a contradiction of this same
doc's own claim that "no framework gets a hand-tuned advantage the
others do not get."
Give Spring Boot an identical second boot under the identical -t4
-c100 -d10s profile, replaying the same trained AOT cache used for its
main run (so it isn't paying a second cold-start penalty CWIST doesn't
pay either). Wire the new spring_tuned_* fields through:
- webserver-result.json output (workflow)
- docs/webserver-benchmark-trends.svg data history (unchanged, reads
from the JSON)
- the README "Tuned low-latency run" callout (scripts/ci/benchmark.py),
now showing both CWIST and Spring Boot side by side instead of CWIST
alone
- docs/webserver-benchmark.md, documenting the methodology so this
number stays reproducible/auditable like the rest of the suite
Axum and Gin are not included in this second pass - noted explicitly
in the doc rather than left implicit, since that's still an asymmetry
in CWIST's favor for those two comparisons (out of scope for this fix,
which was specifically about Spring Boot).
Not runnable/verifiable outside GitHub Actions (needs the JVM/Maven/
Go/Rust toolchains and a real network device the workflow sets up), so
this is reviewed for correctness against the existing (already
CI-proven) CWIST tuned-run code it mirrors, not executed locally.
Confirmed both the workflow YAML and scripts/ci/benchmark.py parse
cleanly (yaml.safe_load / ast.parse).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The web-server benchmark's "tuned low-latency run" (
wrk -t4 -c100, vs the main-t12 -c400profile) was CWIST-only. It got published as a standalone README headline ("Tuned low-latency run: X req/s at Yms") with no Spring Boot equivalent anywhere nearby — so the comparison a reader would naturally make (tuned CWIST vs. the Spring Boot row in the table above) was CWIST's best case against a number Spring was never measured at. This also contradicted the doc's own "no framework gets a hand-tuned advantage the others do not get" claim.Fix
Spring Boot now gets an identical second boot under the identical
-t4 -c100 -d10sprofile, replaying the same trained AOT cache as its main run (no extra cold-start penalty CWIST doesn't also avoid).spring_tuned_*fields flow through the result JSON, and the README "Tuned low-latency run" section now shows CWIST and Spring Boot side by side instead of CWIST alone.docs/webserver-benchmark.mddocuments the methodology.Axum/Gin are explicitly not included in this pass — noted in the doc as a known remaining asymmetry, out of scope here since this fix was specifically about the Spring Boot comparison.
Verification
Not runnable outside GitHub Actions (needs JVM/Maven/Go/Rust toolchains + network setup the workflow provisions), so this was reviewed for correctness against the existing CWIST tuned-run code it mirrors rather than executed locally.
.github/workflows/bsd-kqueue-benchmarks.ymlparses clean withyaml.safe_load,scripts/ci/benchmark.pyparses clean withast.parse.