Add a Kotlin-vs-Rust performance gate - #459
Merged
tinder-maxwellelliott merged 3 commits intoAug 15, 2026
Merged
Conversation
The Rust implementation only earns its place if it is faster than the Kotlin CLI it replaces, but nothing checked that. tools/benchmark.py answers "how much faster on this repository" and needs a real checkout, a Bazel install and Hyperfine, so it cannot run as a gate. tools/perf_gate.py runs both binaries over generated workloads and exits non-zero unless Rust wins: * tools/perf_workload.py builds the fixtures with nothing but the Python standard library -- a synthetic streamed_proto stream of blaze_query Target messages, the workspace those targets name (source and .bzl files, so source and bzl-seed hashing read real bytes), a replay bazel shim answering version/mod/info/query, and hash-file pairs for the diff commands. Same spec, same bytes, any machine; * outputs are compared before timings are reported, so a speedup can never come from the two implementations doing different work; * rounds are interleaved and alternate which binary goes first, so drift on a shared runner penalizes both rather than biasing one; * each workload also reports a startup-adjusted time -- median wall time minus that implementation's own --version median -- because Rust would beat a JVM on process launch no matter how slow its hashing became. That number is gated separately; * defaults require Rust to be faster on the median, faster on the startup-adjusted median, and faster in every paired round. Also wired up: //tools:perf-gate, `make perf-gate`, a Performance Gate workflow on pull requests, and //tools:perf_gate_test in the CI matrix. The gate's unit tests drive the runner with stub binaries whose relative speed the test controls, which is the only way to assert the gate fails when Rust is not faster; they also check the fixture encoder against an independent protobuf decoder and exercise the replay shim. Measured on a 4-core Linux container against bazel-diff 40.0.2: Rust is 8.6x faster on a 24,000-target generate-hashes (7.0x startup-adjusted) and 3.1x faster diffing 150,000 hashed targets (2.5x adjusted), with every workload passing its parity check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019H5EbJUyEw98KHsKZk31we
The job failed with a test exit while the identical target set passed on Bazel 9.x on the same commit; re-triggering to confirm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LE7H84eCeMtCB3aGLT1REn
…aude/rust-kotlin-perf-testing-lg6lwg # Conflicts: # README.md # tools/readme_template.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Rust implementation only earns its place if it is faster than the Kotlin CLI it replaces, but nothing checked that. tools/benchmark.py answers "how much faster on this repository" and needs a real checkout, a Bazel install and Hyperfine, so it cannot run as a gate.
tools/perf_gate.py runs both binaries over generated workloads and exits non-zero unless Rust wins:
Also wired up: //tools:perf-gate,
make perf-gate, a Performance Gate workflow on pull requests, and //tools:perf_gate_test in the CI matrix.The gate's unit tests drive the runner with stub binaries whose relative speed the test controls, which is the only way to assert the gate fails when Rust is not faster; they also check the fixture encoder against an independent protobuf decoder and exercise the replay shim.
Measured on a 4-core Linux container against bazel-diff 40.0.2: Rust is 8.6x faster on a 24,000-target generate-hashes (7.0x startup-adjusted) and 3.1x faster diffing 150,000 hashed targets (2.5x adjusted), with every workload passing its parity check.
Claude-Session: https://claude.ai/code/session_019H5EbJUyEw98KHsKZk31we