think-tool-recovery: model EOS-es after forced `</think>` injection — test fails with both official Flash GGUFs

[nazerim]

./ds4_test --think-tool-recovery fails on current main (b7e9f00). The
recovery mechanism itself still works: detection triggers exactly when the
stanza opening completes (trigger=22/23), and the forced close
("</think>", two tokens) is injected and eval'd into the session. But the
FIRST sampled token after injection is EOS, so the test's final phase —
"verifies the live model then emits a valid, parseable call" (per 91bafb5) —
fails:

tests/ds4_test.c:6396: assertion failed: saw_end
tests/ds4_test.c:6404: assertion failed: calls.len > 0 && ...
ds4-test: think-tool-recovery recovered=1 gen_tokens=2 calls=0 name=-

91bafb5 introduced this on the measurement that "that position predicts a
fresh stanza opening strongly enough that the call restarts cleanly". That
no longer holds for either official Flash GGUF.

(Formatting note: the think open/close and DSML markers in this issue are
written inside quoted/code spans so chat clients that treat them as control
tags render this cleanly.)

Environment

• M5 Max 128 GB, Metal backend
• Models: official antirez/deepseek-v4-gguf releases, sha256-verified
  against HF LFS oids:
  • DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf
    (undated; oid edabc92a…)
  • same name with -0731 suffix (oid 659e22fb…)

Matrix (all cells run on this hardware)

tree	undated GGUF	-0731 GGUF
54b36ed	PASS	FAIL
b7e9f00 (main)	FAIL	FAIL

Failure signature is identical in every failing cell: recovery triggers,
injection succeeds, then immediate EOS.

Interpretation — two independent effects

1. Checkpoint effect: the July-31 weights EOS at the injection point even on
   the OLD engine (54b36ed row: undated passes, -0731 fails). Related: DeepSeek-V4-Flash-0731 support #635.
2. Engine effect: commits in 54b36ed..b7e9f00 shift the older (undated)
   weights into the same EOS behavior (undated column: pass → fail).
   Bisectable using the undated GGUF between those two refs.

Related observations

• DS4_TEST_RECOVERY_PROBE=1 shows the model's NATURAL turn now closes
  "</think>" before emitting the tool call (well-formed list_files stanza).
  The pathological scenario the recovery targets — stanza opening inside an
  unclosed "<think>" block — no longer arises naturally, so the recovery
  rarely fires; but when it does (or when the test's force-fed prefix is
  replayed), the continuation is EOS instead of a restarted call.
• 91bafb5 notes re-emitting the stanza opening was rejected because "the
  duplicated opening reads as an already-made call and the model ends the
  turn" — the model now ends the turn even without the duplicated opening.
• Practical impact is milder than pre-Unclosed </think> before tool call stalls agent session (finish=stop, no tool call delivered) #318: instead of burning to the token
  limit with the call dropped at parse, the turn ends short with no tool
  call (retryable by the agent loop). Agent sessions relying on recovery for
  the pathological case get a dropped call rather than a recovered one.
• PR cuda: implement real per-(layer,expert) LRU for --ssd-streaming-cache-experts #647's test notes independently describe think-tool-recovery as
  "pre-existing flaky … flip[ping] between runs" on CUDA/GB10, suggesting
  the continuation phase is borderline on other hardware too.

Fork-side mitigations (context from a downstream fork)

Our fork (nazerim/ds4) hardens the recovery's AFTERMATH; all of it passes on
the merged tree and is independent of this regression:

• KV-cache consistency: recovery's injected close used to diverge from the
  canonical form the next request renders, causing token-mismatch disk-KV
  misses and full-prefill rebuilds. A recovery-fired flag now forces
  checkpoint canonicalization at turn end (in-place rewrite anchored at the
  longest common prefix; disk prefix-anchor fallback for oversized tails).
• Streaming consistency: THINKING-mode streams hold back DSML markers so
  streamed reasoning matches the canonical KV suffix.
• Detection hardening: scan-from off-by-one fix, 8-byte hold string, plus
  two regression tests (prose-marker immunity, spurious-then-legitimate
  stanzas) — all green on current main.

So detection/injection/state-consistency are sound; only the live-model
continuation assumption regressed.

Repro

make ds4_test
./ds4_test --think-tool-recovery            # with either official GGUF
DS4_TEST_RECOVERY_PROBE=1 ./ds4_test --think-tool-recovery  # natural turn

Happy to help bisect the engine effect if useful.

[rell666]

Corroborating this from the other backend — same failure, CUDA instead of Metal, so it is not
Metal-specific sampling.

Environment: DGX Spark / GB10 (sm_121), CUDA 13.0.88, Linux aarch64. Model is the same
-0731 release you list (…-imatrix-fixed-0731.gguf), verified by quantize.imatrix.file
chunk count 202100 rather than by filename.

Identical fingerprint, on a tree predating yours:

ds4-test: think-tool-recovery trigger=22/23 injected_tokens=2
tests/ds4_test.c:1950: assertion failed: calls.len > 0 && !strcmp(calls.v[0].name, "list_files")
ds4-test: think-tool-recovery recovered=1 gen_tokens=2 calls=0 name=-

Same trigger=22/23, same recovered=1 gen_tokens=2 calls=0 name=-. The assertion line differs
only because the test moved between our tree and b7e9f00.

Worth noting for anyone reading the raw log: the continuation we captured looks like a stray
`</think>` welded inside the opening `<|DSML|tool_calls>` tag, which is what led us to
the wrong root cause. Read through your explanation it is the benign reading — gen_tokens=2 is
just the two injected tokens with nothing sampled after them. Flagging it because that log shape
is misleading if you come at it cold.

A control you may find useful: it is not a local patch. Our tree carries an unrelated
reasoning-effort-tier patch, so we built unpatched 80ebbc39 in a clean worktree on the
same checkpoint and ran the same group. The harness asserts the tree is unpatched first:

base worktree at: 80ebbc3 Merge pull request #319 from rinaldofesta/fix/eval-grader-false-negatives
confirmed unpatched: no DS4_THINK_ULTRA in ds4.c

Result: identical failure, byte for byte — same trigger, same injected token count, same
assertion, same recovered=1 gen_tokens=2 calls=0 name=-. So the defect is present in stock
80ebbc39, i.e. it predates b7e9f00 by a good margin.

One cell of ours disagrees with your matrix, and I would rather surface it than quietly
assume it away. Adding our rows (assuming your undated GGUF is the pre-0731 preview checkpoint,
imatrix chunk count 90042 — please correct me if not):

tree	undated / preview	-0731
54b36ed (yours)	PASS	FAIL
b7e9f00 (yours)	FAIL	FAIL
80ebbc39 stock (ours)	not run	FAIL
80ebbc39 + our effort-tier patch (ours)	FAIL	FAIL

Your 54b36ed + undated passes; our patched 80ebbc39 + preview fails. Three things could
explain that, and I cannot separate them from here:

1. the two checkpoints are not the same file (my assumption above is wrong);
2. the tree difference — 80ebbc39 vs 54b36ed;
3. our effort-tier patch does affect this group on preview specifically, even though it
   demonstrably does not on 0731 (the base-vs-patched control above holds the checkpoint fixed).

The cell that would settle it is stock 80ebbc39 on the preview checkpoint, which neither of
us has run. I can run it on GB10 if that is useful — our box is committed to an unrelated
benchmark for roughly another day, but after that it is a short job.

The checkpoint effect you describe matches what we see, for what it is worth: nothing in the file
metadata distinguishes these two GGUFs except quantize.imatrix.file — general.name,
size_label, file_type, tensor count and byte size are all identical — so it is easy to run the
wrong one and not know. Recording that key on every run saved us more than once.