docs(beat): record what the CI host actually measures for LinReg (0.788/0.805, ~11% headroom) - #2346
Merged
Merged
Conversation
…11% headroom)
Validated the new configuration by dispatching beat-speed-nightly twice onto
the clean-room pool - the host that produced the 1.044 excursion:
apr=265.473ms sklearn=337.106ms ratio=0.788
apr=252.463ms sklearn=313.446ms ratio=0.805
2.2% spread where the old size swung 346% on that same pool, so the stability
goal is met. But the LEVEL differs from my local box and the file said so
incorrectly: apr is ~1.25x faster there, not ~1.85x, leaving ~11% headroom
under the 0.90 ceiling rather than 42%. That host penalises apr more than
sklearn (2.3x vs 1.55x slower than lambda-vector).
Recorded two things a future reader needs. Enlarging the workload again is NOT
the remedy if this fails - apr's advantage shrinks with size, so a bigger
problem makes the ratio worse. And the contract's baseline_floor 0.56 describes
neither this host nor the old size; whether that is slower CI hardware or a
real regression cannot be settled without historical CI data at this size, so
it is left open rather than asserted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2345. This commit existed on that branch but was pushed after the merge queue had already snapshotted the PR, so the squash dropped it — the documented
squash-drops-late-commitstrap, caught by diffing main rather than assuming.Main currently carries a claim I have since measured to be wrong for the host the gate runs on.
What #2345's doc comment says vs. what CI measures
#2345 justified the new workload with numbers from lambda-vector (Threadripper 7960X): apr ~1.85× faster, 42% headroom. I then dispatched
beat-speed-nightlytwice onto the clean-room pool — the host that produced the original 1.044 excursion:The stability goal is met: 2.2% spread, where the 10 000×20 configuration swung 0.234 → 1.044 (346%) on that same pool.
But the level differs materially. On that host apr is ~1.25× faster, not ~1.85×, leaving ~11% headroom under the 0.90 ceiling rather than 42%. It penalises apr more than sklearn — 2.3× vs 1.55× slower than lambda-vector.
Two things recorded for whoever hits this next
Do not enlarge the workload again if this ever fails. apr's advantage shrinks with size (0.26 at 10 000×20 → 0.54 at 200 000×50, locally), so a bigger problem makes the ratio worse. That's the obvious next move and it's the wrong one.
The contract's
baseline_floor: 0.56describes neither host — not this one (0.79–0.81) and not the old size (0.26). Whether that gap is slower CI hardware or a real apr regression cannot be settled without historical CI data at this size, so it is left explicitly open rather than asserted in either direction.Docs only — no code, no threshold changes.
🤖 Generated with Claude Code