Skip to content

fix(beat): the GaussianNB "4.9x" was a dev-box number - re-pin to the host that actually runs it - #2328

Merged
noahgift merged 1 commit into
mainfrom
fix/gnb-beat-host-honest
Jul 28, 2026
Merged

fix(beat): the GaussianNB "4.9x" was a dev-box number - re-pin to the host that actually runs it#2328
noahgift merged 1 commit into
mainfrom
fix/gnb-beat-host-honest

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

PMAT-GNB-SPEED-DEFENSE-001. The beat pinned "Measured ~0.20 (apr ~4.9× faster on a 16-core box)" against a 0.50 ceiling — which reads as 2.5× of headroom. The real headroom on the machine that runs this beat is about 20%, and the gate has been tripping: 07-03 0.563, 07-04 0.520, 07-28 0.601.

Measured this session

host apr sklearn ratio
lambda-vector (modern, 48 threads) ~12.0 ms ~56 ms 0.188 / 0.237 / 0.220 (4.2–5.3×)
mac-server — canonical nightly host, IDLE 28.0 ms 70.3 ms 0.398 (2.51×)
mac-server under nightly load (run logs) 0.278 … 0.601 (straddles 0.50)

This is not decay. git log --since=2026-07-01 -S"GaussianNB" -- crates/aprender-core/src/ is empty; the last impl change was #2181 (2026-06-22) and the ratio has bounced across that entire window. It's a different machine, plus contention.

Why the host moves it so much

apr's GaussianNB is a scalar Rust loop; sklearn's is vectorised numpy. mac-server is a Xeon W-3245 with AVX-512 (avx512f/bw/cd/dq/vl), so numpy closes much of the gap there: lambda → mac-server costs apr 2.3× (12.0 → 28.0 ms) but sklearn only 1.25× (56 → 70.3 ms). On top of that the box runs 17 registered runners on 32 threads, which pushes a busy night the rest of the way over 0.50.

What this does — and does not do

It does NOT weaken the gate. RATIO_CEILING stays 0.50. The idle-host reading (0.398) clears it, so a run over 0.50 remains real signal rather than expected noise. Moving the threshold to make red go away would concede a win we still hold.

It fixes the false baseline, recording each measurement with the host it came from, so nobody re-tightens the gate on a number that machine cannot produce.

It adds a fit/predict phase split, printed every run and included in the failure message — the missing diagnostic. Contention drags both phases up together; an algorithmic regression lands in one. With only a fused total, five red nights told you nothing actionable.

BEAT-SKLEARN-GAUSSIANNB-SPEED: apr=11.675ms sklearn=51.795ms ratio=0.225 (apr 4.44x faster)
BEAT-SKLEARN-GAUSSIANNB-SPEED-PHASES: fit apr=4.666ms skl=15.508ms ratio=0.301
                                    | predict apr=7.078ms skl=37.199ms ratio=0.190

predict is where apr's edge is largest (5.3×) and fit is the weaker half (3.3×) — which also says where the durable fix belongs: SIMD the predict hot loop via trueno to restore real margin on AVX-512 hosts. That remains open under PMAT-GNB-SPEED-DEFENSE-001, deliberately not attempted here because it needs its own before/after on mac-server rather than a dev box.

Note: this beat has been unobservable since 2026-07-26 — the nightly lane died at step 1 on a poisoned uv env cache and fail-fast skipped every later beat (#2326). The 0.601 reading came from re-running the lane after repairing the host.

🤖 Generated with Claude Code

… host that actually runs it

PMAT-GNB-SPEED-DEFENSE-001. The beat pinned "Measured ~0.20 (apr ~4.9x faster on a
16-core box)" against a 0.50 ceiling, which reads as 2.5x of headroom. The real
headroom on the machine that runs this beat is about 20%, and the gate has been
tripping: 2026-07-03 ratio 0.563, 07-04 0.520, 07-28 0.601.

MEASURED THIS SESSION (the 4.9x has never been observed on the CI host):

  lambda-vector (modern 48-thread box):     0.188 / 0.237 / 0.220  (apr 4.2-5.3x)
  mac-server (canonical nightly host, IDLE): 0.398                 (apr 2.51x)
  mac-server under nightly load (run logs):  0.278 .. 0.601        (straddles 0.50)

So this is NOT decay. `git log --since=2026-07-01 -S"GaussianNB" -- crates/aprender-core/src/`
is empty; the last impl change was #2181 on 2026-06-22, and the ratio has been
bouncing across that whole window. It is a different machine, plus contention.

WHY THE HOST MOVES IT SO MUCH. apr's GaussianNB is a scalar Rust loop; sklearn's is
vectorised numpy. mac-server is a Xeon W-3245 with AVX-512 (avx512f/bw/cd/dq/vl), so
numpy closes much of the gap there: moving from lambda-vector to mac-server costs apr
2.3x (12.0ms -> 28.0ms) but sklearn only 1.25x (56ms -> 70.3ms). On top of that the box
runs 17 registered runners on 32 threads, so a busy night pushes it the rest of the way
over 0.50.

WHAT THIS CHANGE DOES - and does NOT do.

It does NOT weaken the gate. RATIO_CEILING stays 0.50. The idle-host reading (0.398)
clears it, so a run over 0.50 remains real signal rather than expected noise. Moving the
threshold to make red go away would concede a win we still hold (see the standing rule
against conceding speed without evidence).

It fixes the false baseline in the doc comment, recording each measurement WITH the host
it came from, so nobody re-tightens the gate on a number that machine cannot produce.

It adds a fit/predict phase split, printed on every run and included in the failure
message. That is the missing diagnostic: contention drags both phases up together, while
an algorithmic regression lands in one. With only a fused total, five red nights told you
nothing actionable. Verified locally:

  BEAT-SKLEARN-GAUSSIANNB-SPEED: apr=11.675ms sklearn=51.795ms ratio=0.225 (apr 4.44x)
  BEAT-SKLEARN-GAUSSIANNB-SPEED-PHASES: fit apr=4.666ms skl=15.508ms ratio=0.301
                                      | predict apr=7.078ms skl=37.199ms ratio=0.190

predict is where apr's edge is largest (5.3x) and fit is the weaker half (3.3x), which
also says where the durable fix belongs: SIMD the predict hot loop via trueno to restore
real margin on AVX-512 hosts. That is the actual defence of this beat and remains open
under PMAT-GNB-SPEED-DEFENSE-001 - deliberately not attempted here, because it needs its
own before/after measurement on mac-server rather than a dev box.

Note this beat has been UNOBSERVABLE since 2026-07-26: the nightly lane died at step 1 on
a poisoned uv env cache and fail-fast skipped every later beat (fixed in #2326). The
0.601 reading above came from re-running the lane after repairing the host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge July 28, 2026 16:01
@noahgift
noahgift added this pull request to the merge queue Jul 28, 2026
Merged via the queue into main with commit e54d3cc Jul 28, 2026
21 of 23 checks passed
@noahgift
noahgift deleted the fix/gnb-beat-host-honest branch July 28, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant