Skip to content

Add a Kotlin-vs-Rust performance gate - #459

Merged
tinder-maxwellelliott merged 3 commits into
Tinder:masterfrom
maxwellE:claude/rust-kotlin-perf-testing-lg6lwg
Aug 15, 2026
Merged

Add a Kotlin-vs-Rust performance gate#459
tinder-maxwellelliott merged 3 commits into
Tinder:masterfrom
maxwellE:claude/rust-kotlin-perf-testing-lg6lwg

Conversation

@tinder-maxwellelliott

Copy link
Copy Markdown
Collaborator

The Rust implementation only earns its place if it is faster than the Kotlin CLI it replaces, but nothing checked that. tools/benchmark.py answers "how much faster on this repository" and needs a real checkout, a Bazel install and Hyperfine, so it cannot run as a gate.

tools/perf_gate.py runs both binaries over generated workloads and exits non-zero unless Rust wins:

  • tools/perf_workload.py builds the fixtures with nothing but the Python standard library -- a synthetic streamed_proto stream of blaze_query Target messages, the workspace those targets name (source and .bzl files, so source and bzl-seed hashing read real bytes), a replay bazel shim answering version/mod/info/query, and hash-file pairs for the diff commands. Same spec, same bytes, any machine;
  • outputs are compared before timings are reported, so a speedup can never come from the two implementations doing different work;
  • rounds are interleaved and alternate which binary goes first, so drift on a shared runner penalizes both rather than biasing one;
  • each workload also reports a startup-adjusted time -- median wall time minus that implementation's own --version median -- because Rust would beat a JVM on process launch no matter how slow its hashing became. That number is gated separately;
  • defaults require Rust to be faster on the median, faster on the startup-adjusted median, and faster in every paired round.

Also wired up: //tools:perf-gate, make perf-gate, a Performance Gate workflow on pull requests, and //tools:perf_gate_test in the CI matrix.

The gate's unit tests drive the runner with stub binaries whose relative speed the test controls, which is the only way to assert the gate fails when Rust is not faster; they also check the fixture encoder against an independent protobuf decoder and exercise the replay shim.

Measured on a 4-core Linux container against bazel-diff 40.0.2: Rust is 8.6x faster on a 24,000-target generate-hashes (7.0x startup-adjusted) and 3.1x faster diffing 150,000 hashed targets (2.5x adjusted), with every workload passing its parity check.

Claude-Session: https://claude.ai/code/session_019H5EbJUyEw98KHsKZk31we

The Rust implementation only earns its place if it is faster than the
Kotlin CLI it replaces, but nothing checked that. tools/benchmark.py
answers "how much faster on this repository" and needs a real checkout,
a Bazel install and Hyperfine, so it cannot run as a gate.

tools/perf_gate.py runs both binaries over generated workloads and exits
non-zero unless Rust wins:

* tools/perf_workload.py builds the fixtures with nothing but the Python
  standard library -- a synthetic streamed_proto stream of blaze_query
  Target messages, the workspace those targets name (source and .bzl
  files, so source and bzl-seed hashing read real bytes), a replay bazel
  shim answering version/mod/info/query, and hash-file pairs for the
  diff commands. Same spec, same bytes, any machine;
* outputs are compared before timings are reported, so a speedup can
  never come from the two implementations doing different work;
* rounds are interleaved and alternate which binary goes first, so drift
  on a shared runner penalizes both rather than biasing one;
* each workload also reports a startup-adjusted time -- median wall time
  minus that implementation's own --version median -- because Rust would
  beat a JVM on process launch no matter how slow its hashing became.
  That number is gated separately;
* defaults require Rust to be faster on the median, faster on the
  startup-adjusted median, and faster in every paired round.

Also wired up: //tools:perf-gate, `make perf-gate`, a Performance Gate
workflow on pull requests, and //tools:perf_gate_test in the CI matrix.

The gate's unit tests drive the runner with stub binaries whose relative
speed the test controls, which is the only way to assert the gate fails
when Rust is not faster; they also check the fixture encoder against an
independent protobuf decoder and exercise the replay shim.

Measured on a 4-core Linux container against bazel-diff 40.0.2: Rust is
8.6x faster on a 24,000-target generate-hashes (7.0x startup-adjusted)
and 3.1x faster diffing 150,000 hashed targets (2.5x adjusted), with
every workload passing its parity check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019H5EbJUyEw98KHsKZk31we
claude added 2 commits August 14, 2026 23:51
The job failed with a test exit while the identical target set passed on
Bazel 9.x on the same commit; re-triggering to confirm.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LE7H84eCeMtCB3aGLT1REn
…aude/rust-kotlin-perf-testing-lg6lwg

# Conflicts:
#	README.md
#	tools/readme_template.md
@tinder-maxwellelliott
tinder-maxwellelliott merged commit 13fe981 into Tinder:master Aug 15, 2026
40 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants