apr qa Golden Output fails on qwen2.5-coder-1.5b-instruct-q4k where it passed 2026-07-30 (observed once — needs a second data point)

[noahgift]

Surfaced by the revived nightly story (#2347). Filing it separately rather than folding it into that PR, and deliberately not calling it a regression yet — I have one observation.

What was seen

Local full run of scripts/qwen-story.sh at 12551dd19 + the #2347 fixes, Beat 2 (apr qa on qwen2.5-coder-1.5b-instruct-q4k.apr):

✗ FAIL Golden Output
  golden_output: Expected one of ["Hello", "Hi", "hey", "hello", "!"],
  got: "I'm sorry, but I'm not sure what you're asking. Could you please
        provide more context or information"

The previous nightly (run 30520528348, 2026-07-30, e514cc5ed) passed the same gate:

✓ PASS Golden Output   2 golden test cases passed        (22.7s)

Why I am not calling it a regression

• No commit between e514cc5ed and 12551dd19 touches inference. That range is feat(pull): apr pull --verify — check the hash pull already records but never compares #2343 (a new apr pull --verify subcommand), fix(ci): qwen-story-daily installed a fresh apr and then ran a 24-day-old one #2344 (shell scripts), fix(beat): the LinReg speed beat was measuring the host, not the algorithms #2345/docs(beat): record what the CI host actually measures for LinReg (0.788/0.805, ~11% headroom) #2346 (a CPU benchmark's constants and docs). None of them are on the decode path.
• My run was on a different host state than the nightly, and the gate went through [GH-480] F2 validation PASSED — launching GPU generation, i.e. GPU sampling.
• I observed it once. The failing case aborted in 4.6s where the passing run took 22.7s for 2 cases, consistent with the first case failing fast.

So the plausible readings are (a) genuine output-quality change, (b) sampling nondeterminism on the GPU path making this gate flaky, or (c) an over-strict golden expectation. I cannot currently distinguish them.

What will settle it

The nightly qwen-story-daily is the confirmation mechanism, and as of #2347 it can actually report its own result again — before that fix it died at Beat 1 and could not have told us either way. The next few runs will show whether this is stable or intermittent.

If it is intermittent, that is the more interesting finding: a golden-output gate that samples on GPU without a pinned seed is a flake generator, and per feedback_main_ci_andon the fix is to make it deterministic, not to #[ignore] it.

Repro

export PATH="$HOME/.cargo/bin:$PATH"
bash scripts/qwen-story.sh          # Beat 2, or:
apr qa ~/models/qwen2.5-coder-1.5b-instruct-q4k.apr

[noahgift]

Reproduced in CI — upgrading this from "observed once".

qwen-story-daily run 30655240128, dispatched manually against main at 3a686ced9 (post-#2347), on the lambda cuda runner — a different host from my local observation:

-- Beat 2: Trust (QA gates) --
✗ FAIL  B2 apr qa  -  no 'ALL GATES PASSED' line
...
13 PASS / 1 FAIL / 1 SKIP        (story complete in 218s, 153 lines, all 8 beats)

So this is now two independent observations on two hosts, which rules out my "(b) local host state" reading. It remains either a genuine output-quality change or a nondeterministic golden gate — still not distinguished, but no longer explainable as a one-off.

Worth noting what the same run proves about the machinery: apr=apr 0.61.0 (3a686ced9) confirms the freshness guard ran a binary built from the commit under test, the story reported exit=2, and the workflow auto-filed #2355. Before #2347 this failure was completely invisible — the story died at Beat 1 and the verdict step was unreachable.

[noahgift]

This is a real regression, not a flaky gate — and it was masked for 24 days.

1. It is deterministic

Three runs (local, dispatched CI, natural nightly) produce a byte-identical completion:

I'm sorry, but I'm not sure what you're asking. Could you please provide more context or information

A nondeterministic sampler would vary. This rules out option (b) from the original report.

2. #2344 did not cause it — it revealed it

Direct A/B on the same machine, same model file (qwen2.5-coder-1.5b-instruct-q4k.apr, untouched since 2026-03-08):

binary	Golden Output	time
apr 0.60.0 (2026-07-06)	✓ PASS — 2 golden cases	20.9s
apr 0.62.0 (341875412)	✗ FAIL	4.5s

Before #2344, qwen-story-daily ran a 24-day-old apr 0.60.0 from ~/.local/bin that shadowed the freshly installed binary. So the 2026-07-30 "PASS" was stale code passing. Fixing the shadowing is what made this visible; the defect landed weeks earlier.

That also corrects the framing in my previous comment: the useful window is not the 4 commits before the first red nightly, it is v0.60.0 → v0.62.0.

3. Bisected

git bisect over the 42-commit range, predicate = apr qa <model> reporting PASS Golden Output, constant build flags (--release --features cuda):

GOOD  fe01cc06c   v0.60.0
GOOD  a93cfe62b
GOOD  c4be392d3   #2322  stop swallowing the CUDA forward failure
BAD   33523d9bc   #2325  "82 workspace crates" is false
BAD   341875412   HEAD

That clears #2322 and narrows the culprit to #2323 (fix(cuda): default Q4K to fp32 MWV on ALL GPUs) or #2324 (CI-only beat gating). #2323 is the only one that changes what the GPU computes. Confirmation in progress.

If it is #2323

That commit is correct — HwDp4a's INT8 activation quant fails the F2 cosine floor (0.9186 < 0.95) and made CUDA fall back to CPU at ~20 tok/s. But it changed decode numerics on every GPU, and this gate is the first thing to notice. The question then becomes whether MWV output on this Q4K model is acceptable-but-different, or actually degraded — which the golden gate alone cannot answer.

Related: the same commit is why the ollama decode claim was withdrawn (#2349).

[noahgift]

Bisected and confirmed: first bad commit is #2323

GOOD  c4be392d3   #2322  stop swallowing the CUDA forward failure
BAD   01524c76e   #2323  default Q4K to fp32 MWV on ALL GPUs

Predicate: apr qa <model> reporting PASS Golden Output, constant build flags across every revision.

Single-binary confirmation

The bisect result is corroborated without rebuilding anything — same apr 0.62.0, same model, only the kernel env var differs:

kernel	Golden Output	time
Mwv (default, post-#2323)	✗ FAIL	5.1s
HwDp4a (HW_DP4A_Q4K=1, pre-#2323)	✓ PASS, 2 cases	25.0s

The Q4K GPU kernel selection is unambiguously the discriminator.

The inversion worth reading twice

HwDp4a is the degraded kernel — it fails F2's first-token cosine floor (0.9186 < 0.95). It "passes" this gate only because F2 rejects it, so the run falls back to CPU and the CPU produces correct output. That is what the 25s vs 5.1s is telling us.

Mwv is the numerically better kernel — F2 accepts it at 0.9937 — and it produces semantically wrong output:

I'm sorry, but I'm not sure what you're asking...

So the kernel that clears the correctness gate generates worse text than the one that fails it.

Scope: narrower than it looks

apr run on the same model is fine on both CPU and GPU:

GPU: "Hello! How can I assist you today?"
CPU: "Hello! How can I assist you today?"

The golden gate goes through generate_gpu_resident (output_verification.rs:465), a different entry point from the one apr run uses. So the defect is confined to the GPU-resident decode path, and the user-facing apr run path is unaffected. #2323 remains the right change — it fixed a ~20 tok/s CPU-fallback collapse — and should not be reverted.

This is exactly what rank 3 exists to catch

F2 validates one token. It passes here, and the generation diverges afterwards. That is the open roadmap item PMAT-F2-DECODE-PHASE-001 ("≥8 decode steps through the production prefill mode") — a first-token cosine check cannot see a decode that goes wrong at step 5.

Had rank 3 been implemented before #2323, this would have been caught in that PR instead of surfacing 24 days later through a stale-binary accident.

Next

1. Root-cause the generate_gpu_resident + Mwv divergence (why does correct-at-token-1 decode wrong after?).
2. Implement rank 3 so the class is gated rather than discovered.

Retitling this from "needs a second data point" — it has three, a bisect, and a single-binary A/B.

[noahgift]

Correction: the discriminator is GPU vs CPU, not HwDp4a vs Mwv

Same binary (apr 0.62.0), same model, same golden prompts — only GPU availability differs:

default (GPU used)              -> Golden Output FAIL
CUDA_VISIBLE_DEVICES="" (CPU)   -> Golden Output PASS, 2/2 cases, 22.6s

My previous comment read the HW_DP4A_Q4K=1 pass as "the old kernel is fine". That was wrong, and the timing was the clue I under-weighted: HwDp4a fails F2 (cosine 0.9186 < 0.95), so setting it makes CUDA get rejected and the run finishes on CPU. Both "passing" configurations were CPU. The 25.0s vs 5.1s split was CPU-vs-GPU all along.

What that changes

The causal chain is not "#2323 chose a worse kernel". It is:

	before #2323	after #2323
sm_89 Q4K default	HwDp4a	Mwv
F2 first-token cosine	0.9186 → REJECT	0.9937 → ACCEPT
what actually decoded	CPU fallback	GPU
golden output	correct	wrong

#2323 did not introduce the defect — it made the GPU decode path reachable, and that path was already producing wrong text. The bug was hidden because F2 had been rejecting GPU entirely on this hardware.

That is a materially larger blast radius than I stated earlier, and I want to be explicit that I got it wrong: I wrote that apr run works on GPU so "the defect is confined to a secondary path". apr run --gpu does produce correct text for an auto-wrapped "Hello", but that is a different token sequence from the gate's pre-formatted ChatML, and it does not license the conclusion I drew. What is established is narrower and worse: GPU decode diverges from CPU decode on at least some real inputs.

Still open

• Which stage diverges. F2 passes at token 1 (0.9937) and the output is wrong by the end, so the divergence is in the decode, not the prefill.
• Whether generate_with_cache (the APR golden path, golden_output.rs:34) and generate_gpu_resident (apr_inference.rs:89) diverge differently.
• Whether GGUF is affected, or only the .apr Q4K path.

This is rank 3, exactly

PMAT-F2-DECODE-PHASE-001 exists because F2 validates one token. Here it passes at 0.9937 and the generation is wrong regardless. A first-token cosine gate is structurally incapable of catching this, and extending it to ≥8 decode steps through the production prefill mode would have failed #2323's PR instead of surfacing 24 days later via a stale-binary accident.

Raising priority: this is a GPU correctness defect on the default path for sm_89, not a flaky gate.

[noahgift]

Blast radius bounded: it is the .apr GPU path, not GPU decode in general

Same binary, three configurations:

format	device	Golden Output	time
GGUF qwen2.5-coder-1.5b-instruct-q4_k_m.gguf	GPU	✓ PASS 2/2	8.2s
APR qwen2.5-coder-1.5b-instruct-q4k.apr	GPU	✗ FAIL	5.1s
APR (same file)	CPU	✓ PASS 2/2	22.6s

GGUF decodes correctly on the GPU at 8.2s, so this is not "GPU decode is broken" — my previous comment left that open and it is now closed. The defect is specific to the APR-format GPU path.

That matters for where to look: GGUF goes through cli/gguf_inference_gpu.rs:133, while .apr goes through cli/apr_inference.rs:89. Both call generate_gpu_resident, so the divergence is more likely in how the APR path reaches it — weight layout, quantised tensor handoff, or cache setup — than in the shared generation routine itself.

Worth noting: the APR file predates all of this (unchanged since 2026-03-08), and the same file decodes correctly on CPU. So the weights are fine; something in the APR→GPU upload or kernel-selection path is not.

(The GGUF run does report GATES FAILED overall, but on a different gate — its Golden Output is clean.)

Suggested next step for whoever picks this up

Compare CPU and GPU logits for the APR model on the exact failing prompt
(<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n) step by step. F2 already
tells us they agree at token 1 (cosine 0.9937), so the first disagreeing step is the
signal — that is precisely the measurement PMAT-F2-DECODE-PHASE-001 (rank 3) is
specified to make routine.

[noahgift]

Correcting my own description of F2 — it is stronger than I said, and that sharpens the hypothesis

I wrote twice above that "F2 validates one token". That is wrong. Reading inference_result.rs:445-465, F2 forwards the whole prompt on both backends and checks every position:

ACCEPT <=> for all p >= 1: argmax(cpu[p]) == argmax(gpu[p])
           AND min_{p>=1} cosine(cpu[p], gpu[p]) >= 0.95
           AND no NaN / zero-norm

Position 0 is deliberately excluded as a benign near-tie. Its own header even explains that a last-token-only gate would miss mid-sequence divergence — the exact mistake I attributed to it.

So F2 is a full prefill-parity gate, and it passes here at 0.9937 across all real positions.

Which makes the hypothesis much more specific

Prefill agrees at every position. The generated text is wrong. The difference between those regimes is the KV cache: prefill computes all positions in one forward; decode steps autoregressively through the cache.

That points at cached-decode corruption on the APR GPU path — the same class as GH-1864 / PMAT-810 ("CertainlyCertainly") — rather than kernel numerics. It also explains every observation cleanly:

• GGUF + GPU passes -> its cache path is exercised constantly and is fine
• APR + CPU passes -> CPU cached decode is fine
• APR + GPU fails -> APR-specific GPU cache setup/handoff
• F2 passes -> prefill never exercises the decode cache the same way

What rank 3 should therefore assert

PMAT-F2-DECODE-PHASE-001 is not "extend F2 to more tokens" — F2 already covers the prompt. It is: carry the CPU/GPU comparison >=8 steps into autoregressive decode, through the production prefill mode and the real cache, which is the regime F2 structurally cannot reach.

Same lesson as #2337, where format_parity compared one prefill argmax until it was made 64 teacher-forced steps through the cache path — and found a real defect immediately.

Apologies for the earlier mischaracterisation; the gate's authors got this right and I misread it.

[noahgift]

Reproduction boundary: it does NOT reproduce through apr run

I tried to reproduce the failure outside apr qa. It does not reproduce, and that is a useful constraint rather than a dead end.

Same binary, same .apr model, GPU vs CPU (CUDA_VISIBLE_DEVICES=""):

prompt via apr run	GPU	CPU	agree?
"Hello"	Hello! How can I assist you today?	identical	yes
long multi-part prompt (~30 tok)	Whispers secrets to the shore, / Nature's symphony.	identical	yes
the gate's literal ChatML text	Hello! How can I assist you today?	identical	yes

So: not prompt length, not ChatML content, not GPU-in-general. apr qa on the same model and the same logical prompt fails on GPU and passes on CPU.

What that isolates

apr run and the golden gate reach generation differently:

• gate (golden_output.rs:25-34): tokenizer.encode(prompt) on an already-formatted ChatML string, then .with_input_tokens(prompt_tokens) → generate_with_cache(...). It deliberately bypasses ChatML auto-wrap — the code comment says so explicitly, to avoid double-wrapping.
• apr run: auto-wraps, so feeding it the literal ChatML above actually produces a double-wrapped sequence.

Both wrapped forms decode correctly on GPU. The gate's single-wrapped, raw-encoded sequence does not. So the trigger is tied to the gate's entry path — generate_with_cache with caller-supplied input_tokens — and not to APR+GPU generally.

Also checked, so it is not a confound

apr run prints (cached) on every run, which I initially suspected was a response cache that would have invalidated all the comparisons above. It is not: run_entry.rs:334-343 prints (cached) vs (downloaded) for model acquisition. No inference-output caching is involved.

Narrowed next step

Instrument the gate's own call — generate_with_cache with explicit input_tokens — and dump per-step CPU vs GPU logits for
<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n. F2 already proves prefill parity at 0.9937 across all positions, so the first disagreeing decode step is the answer. Reproducing through apr run is a dead end; the harness has to call the same entry point the gate does.

[noahgift]

Final narrowing — everything else is ruled out

Reading output_verification.rs:505 golden_output_apr, the gate does:

let tokenizer = apr_model.load_embedded_bpe_tokenizer()...;   // EMBEDDED tokenizer
let prompt_tokens = tokenizer.encode(prompt);                 // raw encode of ChatML text
let config = InferenceConfig::new(path)
    .with_input_tokens(prompt_tokens)                         // bypasses prepare_tokens auto-wrap
    .with_max_tokens(max_tokens)
    .with_temperature(0.0)
    .with_top_k(1);
run_inference(&config)

Ruled out by measurement, in order:

candidate	verdict
GPU decode broken in general	no — GGUF + GPU passes (8.2s)
Kernel choice (HwDp4a vs Mwv)	no — HwDp4a "passes" only via CPU fallback
Prompt length	no — a ~30-token prompt agrees on both
ChatML content	no — the literal ChatML text agrees on both via apr run
Sampling config	no — apr run already defaults to temperature 0.0, top_k 1, identical to the gate
Response caching confound	no — (cached) is model acquisition, not inference
Model file drift	no — unchanged since 2026-03-08, and passes on CPU

What remains is the one thing apr run cannot exercise: with_input_tokens carrying a sequence produced by the APR file's embedded BPE tokenizer, fed straight to run_inference without prepare_tokens.

The specific thing to check first

Does apr_model.load_embedded_bpe_tokenizer().encode("<|im_start|>...") emit the special-token IDs, or the literal characters <, |, im_start, |, >? apr run's auto-wrap path almost certainly injects the correct special IDs directly, which would explain why every apr run variant agrees while the gate's sequence does not — they are not the same token sequence at all.

If the embedded tokenizer is emitting literal text, then the gate is feeding the model a malformed prompt, and the real defect is that the GPU path mishandles it while CPU happens to recover — which is still a GPU bug, but a different one from "cached decode is corrupt", and it would also mean the gate has been asserting on a prompt it did not intend since it was written.

Dumping both token sequences (gate path vs apr run path) for the same text is a ~10-line diagnostic and would settle it immediately. I am stopping here rather than guessing further.

[noahgift]

Reversing my diagnosis: this is a brittle GATE, not a GPU correctness defect

I escalated this as "a wrong-output defect on the default sm_89 path" and said I would not cut a release over it. That was wrong. Here is the measurement that settles it.

First, a flaw in my own earlier experiments: I built the harness with --features inference but not --features cuda, and labelled the runs "GPU" from an env-var proxy. Those runs were CPU. Everything I reported from them ("with_input_tokens is correct on GPU", "passes in sequence", "passes at max_tokens=512") was mislabelled. With --features cuda the CUDA path engages (a [GH-480-TRACE] try_apr_cuda_inference line appears, and case 1's output shifts from "2+2 equals 4." to "2 + 2 equals 4.").

With CUDA genuinely enabled, five prompts, same binary, same config (temperature 0.0, top_k 1, max_tokens 512):

#	prompt	CPU	CUDA
1	What is 2+2?	2+2 equals 4.	2 + 2 equals 4.
2	Hello	Hello! How can I assist you today?	I'm sorry, but I'm not sure what you're asking...
3	Hi	I'm sorry, but I'm not sure what you're asking...	I'm here to help! How can I assist you today?
4	Hello there, how are you doing today my friend?	Hello! I'm doing well, thank you.	same
5	What is the capital of France?	The capital of France is Paris.	same

Look at row 3. The exact "I'm sorry, but I'm not sure what you're asking" string that this issue is about appears on CPU as well — just for Hi instead of Hello. Both backends produce that evasive completion for bare one-word greetings; they merely disagree about which greeting lands on which side.

What is actually happening

The model's response to a minimal greeting is borderline. Under greedy decoding (top_k=1), a near-tie at an early position resolves to a different token depending on tiny numerical differences — and CPU vs CUDA kernels differ well within tolerance (F2 measures full prefill parity at cosine 0.9937 across every position, with no argmax mismatch). The continuation then diverges into a different, but entirely coherent, answer.

No corruption, no garbage, no cache bug. Substantive prompts (1, 4, 5) agree on both backends.

So the defect is the gate

golden_test_cases() case 2 asserts on the exact content of a greedy continuation from a 9-token prompt sitting on a knife-edge. That is a coin-flip across backends, builds, and kernel changes — which is precisely why it flipped when #2323 made the CUDA path reachable. It will keep flapping.

Recommended fix: replace the bare-greeting case with a prompt whose greedy continuation is stable. Cases 1, 4 and 5 above are all backend-stable and still exercise real generation; What is the capital of France? → Paris is a better golden than Hello → Hello.

Corrections to my earlier comments on this issue

• "GPU decode diverges from CPU on real inputs" — overstated. They diverge on one near-tie prompt and agree on substantive ones.
• "cached-decode corruption, Qwen2.5-7B Q4_K GPU inference produces gibberish — 'ampiezza' (wgpu) / '<|im_start|>' (cuBLAS) — regression vs #374 / #559 #1864 class" — no evidence; withdrawn.
• "I would not cut v0.63.0 until this is resolved" — withdrawn. This is not release-blocking. It is a flaky gate that should be de-flaked.

What stands: #2323 exposed it rather than caused it, and the 24-day masking by a stale binary was real. The reproduction harness is in crates/apr-cli/tests/golden_prompt_tokenization.rs and shows the tokenizer is clean (9 tokens, correct control ids, exact round-trip).