bench/results: cross-architecture verification on x86-64 (EPYC 9124) - #218
Merged
Conversation
The Air's canonical table is one arm64 machine. This is the same nine legs on a second architecture — AMD EPYC 9124, Ubuntu 24.04, kernel 6.8.0-90 — with schema pinned at d398ec4 and every serialize runtime checked out to the exact commit the Air run used, so architecture is the variable. corpus_id 6b213fbfa1a03a99 on all nine legs, identical to the Air: the wire is byte-identical across architectures. ORDER Air: c cpp rust java go cs dart js elixir x86: cpp c java rust go cs dart js elixir Two reshuffles, both at the top; the tail (go, cs, dart, js, elixir) holds its order exactly. 1. The c/cpp tie does NOT survive the crossing. Air c=98% (cpp-relative, inside noise). x86 c=111% under gcc 13.3 and 107% under a clang 18 control leg run in the same window — c is slower than cpp on x86 under BOTH compilers, at 15-50x the measured spread. Architecture is the primary term; the compiler adds ~4 points. 2. rust/java swap. Air rust 154 < java 157; x86 java 121 < rust 126. Both compress hard toward cpp on x86 (-28 and -36 points). go is the stability result: 239% -> 233%, a 6-point move across an architecture change. js (+30) and elixir (+125) are the only legs that lose ground on x86. Two independent full sweeps in the same window agree within 1 point on every leg except elixir (13). The clang-18 control leg is committed beside the primary CSV: clang beats gcc on this box by 21% (cpp) and 25% (c), which is why the c/cpp finding is reported against both. Measured with the game service resident (pre-release, no players, owner confirmed) — one pinned game worker thread per isolated core at ~18% CPU including bench core 15. Recorded in each CSV's noise line. The load is uniform across legs, which is what an ordering/ratio comparison tolerates; the absolute rates are correspondingly conservative. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cross-architecture verification of the canonical nine-language table. Do not merge yet — this is a findings PR; the c/cpp tie ruling is affected and that is the owner's call.
The two tables
round_trip, best-of-3 max rate, cpp = 100% denominator (run.sh's own blender). Higher = slower.corpus_id 6b213fbfa1a03a99on all nine legs, identical to the Air — the wire is byte-identical across architectures. That gate passed before any timing was trusted.Finding 1 — the c/cpp tie does not survive the crossing
The tie ruling predicts c and cpp stay inside noise. On x86 they do not:
c is slower than cpp on x86 under both compilers, at 15-50x the measured spread. I ran the clang-18 control specifically so this could not be dismissed as a gcc artifact — architecture is the primary term, the compiler adds roughly 4 points on top. Both control CSVs are committed.
Side result: on this box clang beats gcc by +20.6% (cpp) and +25.2% (c) on round_trip. The primary sweep uses the box default (gcc), which is the conservative choice.
Finding 2 — rust/java swap
Air
rust 154 < java 157; x86java 121 < rust 126. Both compress hard toward cpp on x86 (rust -28, java -36 points). Reproduced in two independent sweeps. Caveat: java carries the highest spread in the run (6.53% round_trip, JIT warmup) — still well inside the 15% NOISY threshold, and its max rate was stable across both sweeps.Finding 3 — go is the stability result
239% → 233%. A 6-point move across an architecture change, a different compiler backend and a different inliner. The 239% figure is a property of the emitted code, not of the M2.
js (+30) and elixir (+125) are the only legs that lose ground on x86.
Ratios and magnitude
Magnitude belongs to the machine, as expected — the EPYC runs every leg slower in absolute terms (cpp 3.476M → 2.074M msg/s, 60%). The per-language x86/Air rate ratios spread from 52% (c) to 77% (java), and that spread is the reshuffle.
Method
d398ec4(the Air's canonical commit). Main has since advanced tof95a683(elixir elixir #174: lever M — one binary construction per barrier #207 lever M), which changes generated elixir — the elixir row here is the d398ec4 elixir, deliberately, so the comparison is apples-to-apples.cebaed2/37db942/ae730d5/6b52aa6/f08327e/e391f1e).taskset -c 15, builds on cores 3-14, nice 10. Core 0 never touched.Measurement conditions, stated plainly
The game service was resident throughout (pre-release, no players, owner-confirmed), with one pinned worker thread per isolated core at ~18% CPU including bench core 15. This is recorded in each CSV's
noiseline. The load is uniform across all nine legs, which is what an ordering/ratio comparison tolerates; absolute rates are correspondingly conservative. I did not stop the service.Harness bug spotted
bench/run.sh:79setsOUT=""unconditionally, which clobbers an inheritedOUTenv var — soOUT=... bench/run.shsilently writes to the default path instead. Only the--outflag works. It cost me one overwritten sweep. Worth either honouring the env var or documenting that it is flag-only.