fix(qa): the Golden Output gate was asserting on a coin flip, not on correctness - #2359
Merged
Conversation
…correctness `apr qa` Golden Output has failed on GPU and passed on CPU since #2323, and the nightly has been red for it (#2350). It is NOT a GPU defect. The gate picked a prompt whose greedy continuation sits on a near-tie, so it flips between backends. MEASURED, one binary, one model, greedy (temperature 0.0 / top_k 1), 512 tokens: prompt CPU CUDA --------------------------------- ----------------------- ----------------------- "Hello" "Hello! How can I..." "I'm sorry, but I'm not sure what you're asking" "Hi" "I'm sorry, but I'm not "I'm here to help!..." sure what you're asking" "What is 2+2?" "2+2 equals 4." "2 + 2 equals 4." "Hello there, how are you doing "Hello! I'm doing well, same today my friend?" thank you." "What is the capital of France?" "The capital of France same is Paris." The decisive row is the second: the SAME evasive completion this issue is named after appears on CPU, just for "Hi" instead of "Hello". Both backends answer bare one-word greetings evasively; they only disagree about which one tips over. Substantive prompts agree on both. So this is small legitimate kernel numerics (F2 measures full prefill parity at cosine 0.9937 with ZERO argmax mismatches) amplified by greedy decoding at a near-tie. #2323 did not break decoding — it made the CUDA path reachable on sm_89, and the coin landed the other way. FIX: replace the bare greeting with two wide-margin prompts, both verified to produce identical continuations on CPU and CUDA. The gate now runs three cases instead of two, so this strengthens rather than weakens it. Verified end to end on GPU with a HEAD-built binary: "PASS Golden Output 3 golden test cases passed", ALL GATES PASSED. RATCHET: `golden_prompts_are_not_bare_one_word_messages` rejects any golden prompt whose user message is under three words. Word count is a crude proxy for "wide argmax margin", but it is checkable without a GPU in CI and it blocks the exact shape that cost 24 days of red nightly. Mutation-verified: reinstating "Hello" turns it RED naming the prompt and the word count. ALSO: scripts/apr_bin.sh derived the checkout from `${BASH_SOURCE[0]}`, which is BASH-ONLY. Sourcing it from zsh (the dev box's interactive shell) left it empty, so `dirname ""` gave `.`, the `cd ..` escaped the checkout, and `cargo metadata` reported a DIFFERENT workspace's target dir — the orphaned /mnt/nvme-raid0/targets/aprender. A resolver that silently resolves against the wrong workspace is precisely what that file exists to prevent. Now uses `git rev-parse --show-toplevel`, which is portable and worktree-correct. Verified in both shells; bash resolves the fresh binary, zsh fails closed. Adds crates/apr-cli/tests/golden_prompt_tokenization.rs — the harness used to reach all of the above. It also proves the embedded BPE tokenizer is clean (9 tokens, correct 151644/151645 control ids, exact round-trip), which ruled out the "gate feeds malformed text" hypothesis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the long-running #2350 red nightly. It is not a GPU defect — the gate picked a prompt whose greedy continuation sits on a near-tie, so it flips between backends.
Measured — one binary, one model, greedy (
temperature 0.0,top_k 1), 512 tokensHelloHello! How can I assist you today?I'm sorry, but I'm not sure what you're askingHiI'm sorry, but I'm not sure what you're askingI'm here to help! How can I assist you today?What is 2+2?2+2 equals 4.2 + 2 equals 4.Hello there, how are you doing today my friend?Hello! I'm doing well, thank you.What is the capital of France?The capital of France is Paris.Read row 2. The same evasive completion this issue is named after appears on CPU — just for
Hiinstead ofHello. Both backends answer bare one-word greetings evasively; they only disagree about which one tips over. Substantive prompts agree on both.So this is small, legitimate kernel numerics — F2 measures full prefill parity at cosine 0.9937 with zero argmax mismatches — amplified by greedy decoding at a near-tie. #2323 didn't break decoding; it made the CUDA path reachable on sm_89 and the coin landed the other way.
Fix
Replace the bare greeting with two wide-margin prompts, both verified to produce identical continuations on CPU and CUDA. The gate now runs three cases instead of two — this strengthens it rather than weakening it.
Verified end-to-end on GPU with a HEAD-built binary:
Ratchet
golden_prompts_are_not_bare_one_word_messagesrejects any golden prompt whose user message is under three words. Word count is a crude proxy for "wide argmax margin", but it's checkable without a GPU in CI and blocks the exact shape that cost 24 days of red nightly.Mutation-verified: reinstating
"Hello"turns it RED, naming the prompt and the word count.Also fixed:
apr_bin.shwas bash-onlyIt derived the checkout from
${BASH_SOURCE[0]}. Sourced from zsh (the dev box's interactive shell) that is empty, sodirname ""→., thecd ..escaped the checkout, andcargo metadatareported a different workspace's target dir — the orphaned/mnt/nvme-raid0/targets/aprender.A resolver that silently resolves against the wrong workspace is exactly what that file exists to prevent. Now uses
git rev-parse --show-toplevel: portable and worktree-correct. Verified in both shells — bash resolves the fresh binary, zsh fails closed.Harness
Adds
crates/apr-cli/tests/golden_prompt_tokenization.rs, used to reach all of the above. It also proves the embedded BPE tokenizer is clean (9 tokens, correct151644/151645control ids, exact round-trip), which ruled out the "gate feeds malformed text" hypothesis.🤖 Generated with Claude Code