Skip to content

fix(qa): the Golden Output gate was asserting on a coin flip, not on correctness - #2359

Merged
noahgift merged 1 commit into
mainfrom
fix/golden-case-near-tie
Aug 1, 2026
Merged

fix(qa): the Golden Output gate was asserting on a coin flip, not on correctness#2359
noahgift merged 1 commit into
mainfrom
fix/golden-case-near-tie

Conversation

@noahgift

@noahgift noahgift commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Closes the long-running #2350 red nightly. It is not a GPU defect — the gate picked a prompt whose greedy continuation sits on a near-tie, so it flips between backends.

Measured — one binary, one model, greedy (temperature 0.0, top_k 1), 512 tokens

prompt CPU CUDA
Hello Hello! How can I assist you today? I'm sorry, but I'm not sure what you're asking
Hi I'm sorry, but I'm not sure what you're asking I'm here to help! How can I assist you today?
What is 2+2? 2+2 equals 4. 2 + 2 equals 4.
Hello there, how are you doing today my friend? Hello! I'm doing well, thank you. same
What is the capital of France? The capital of France is Paris. same

Read row 2. The same evasive completion this issue is named after appears on CPU — just for Hi instead of Hello. Both backends answer bare one-word greetings evasively; they only disagree about which one tips over. Substantive prompts agree on both.

So this is small, legitimate kernel numerics — F2 measures full prefill parity at cosine 0.9937 with zero argmax mismatches — amplified by greedy decoding at a near-tie. #2323 didn't break decoding; it made the CUDA path reachable on sm_89 and the coin landed the other way.

Fix

Replace the bare greeting with two wide-margin prompts, both verified to produce identical continuations on CPU and CUDA. The gate now runs three cases instead of two — this strengthens it rather than weakening it.

Verified end-to-end on GPU with a HEAD-built binary:

✓ PASS Golden Output   3 golden test cases passed   (7.4s)
✓ ALL GATES PASSED

Ratchet

golden_prompts_are_not_bare_one_word_messages rejects any golden prompt whose user message is under three words. Word count is a crude proxy for "wide argmax margin", but it's checkable without a GPU in CI and blocks the exact shape that cost 24 days of red nightly.

Mutation-verified: reinstating "Hello" turns it RED, naming the prompt and the word count.

Also fixed: apr_bin.sh was bash-only

It derived the checkout from ${BASH_SOURCE[0]}. Sourced from zsh (the dev box's interactive shell) that is empty, so dirname ""., the cd .. escaped the checkout, and cargo metadata reported a different workspace's target dir — the orphaned /mnt/nvme-raid0/targets/aprender.

A resolver that silently resolves against the wrong workspace is exactly what that file exists to prevent. Now uses git rev-parse --show-toplevel: portable and worktree-correct. Verified in both shells — bash resolves the fresh binary, zsh fails closed.

Harness

Adds crates/apr-cli/tests/golden_prompt_tokenization.rs, used to reach all of the above. It also proves the embedded BPE tokenizer is clean (9 tokens, correct 151644/151645 control ids, exact round-trip), which ruled out the "gate feeds malformed text" hypothesis.

🤖 Generated with Claude Code

…correctness

`apr qa` Golden Output has failed on GPU and passed on CPU since #2323, and the
nightly has been red for it (#2350). It is NOT a GPU defect. The gate picked a
prompt whose greedy continuation sits on a near-tie, so it flips between
backends.

MEASURED, one binary, one model, greedy (temperature 0.0 / top_k 1), 512 tokens:

  prompt                             CPU                      CUDA
  ---------------------------------  -----------------------  -----------------------
  "Hello"                            "Hello! How can I..."    "I'm sorry, but I'm not
                                                               sure what you're asking"
  "Hi"                               "I'm sorry, but I'm not  "I'm here to help!..."
                                      sure what you're asking"
  "What is 2+2?"                     "2+2 equals 4."          "2 + 2 equals 4."
  "Hello there, how are you doing    "Hello! I'm doing well,  same
   today my friend?"                  thank you."
  "What is the capital of France?"   "The capital of France   same
                                      is Paris."

The decisive row is the second: the SAME evasive completion this issue is named
after appears on CPU, just for "Hi" instead of "Hello". Both backends answer
bare one-word greetings evasively; they only disagree about which one tips over.
Substantive prompts agree on both.

So this is small legitimate kernel numerics (F2 measures full prefill parity at
cosine 0.9937 with ZERO argmax mismatches) amplified by greedy decoding at a
near-tie. #2323 did not break decoding — it made the CUDA path reachable on
sm_89, and the coin landed the other way.

FIX: replace the bare greeting with two wide-margin prompts, both verified to
produce identical continuations on CPU and CUDA. The gate now runs three cases
instead of two, so this strengthens rather than weakens it. Verified end to end
on GPU with a HEAD-built binary: "PASS Golden Output 3 golden test cases
passed", ALL GATES PASSED.

RATCHET: `golden_prompts_are_not_bare_one_word_messages` rejects any golden
prompt whose user message is under three words. Word count is a crude proxy for
"wide argmax margin", but it is checkable without a GPU in CI and it blocks the
exact shape that cost 24 days of red nightly. Mutation-verified: reinstating
"Hello" turns it RED naming the prompt and the word count.

ALSO: scripts/apr_bin.sh derived the checkout from `${BASH_SOURCE[0]}`, which is
BASH-ONLY. Sourcing it from zsh (the dev box's interactive shell) left it empty,
so `dirname ""` gave `.`, the `cd ..` escaped the checkout, and `cargo metadata`
reported a DIFFERENT workspace's target dir — the orphaned
/mnt/nvme-raid0/targets/aprender. A resolver that silently resolves against the
wrong workspace is precisely what that file exists to prevent. Now uses
`git rev-parse --show-toplevel`, which is portable and worktree-correct.
Verified in both shells; bash resolves the fresh binary, zsh fails closed.

Adds crates/apr-cli/tests/golden_prompt_tokenization.rs — the harness used to
reach all of the above. It also proves the embedded BPE tokenizer is clean
(9 tokens, correct 151644/151645 control ids, exact round-trip), which ruled out
the "gate feeds malformed text" hypothesis.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge August 1, 2026 11:21
@noahgift
noahgift added this pull request to the merge queue Aug 1, 2026
Merged via the queue into main with commit 46f6e49 Aug 1, 2026
12 checks passed
@noahgift
noahgift deleted the fix/golden-case-near-tie branch August 1, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant