fix(gemma4): `<|channel>thought<channel|>` channel header leaks into text output (not recognized as reasoning)

[pekkah]

Summary

When driving Gemma 4 12B (QAT q4_0) through an in-process agentic tool-call loop (record the structured assistant tool_calls turn + adapter.RenderToolResult(...), then re-infer), the model's answer on the post-tool-result pass is wrapped in a Gemma channel header that leaks into the user-facing GenerateChunkKind.Text stream:

<|channel>thought<channel|>The Program.cs file reads the following environment variables: ...

The markers use the same asymmetric-pipe convention as the tool markers — open <|channel>, close <channel|> — with a channel label (thought) between them, followed by the actual content.

Observed on every pass that re-infers after a tool result (read_file summary, write_file confirmation, edit_file confirmation). The chat template was rendered with enable_thinking=false. Normal (non-tool) single-pass replies do not show this, so it surfaces specifically once history contains an assistant tool_calls turn + a {role:tool} result and the model is asked to continue.

Root cause (hypothesis)

InferenceEngine's reasoning split is keyed on the <think> / </think> token IDs (thinkTokenId/endThinkTokenId). Gemma 4 emits a <|channel>label<channel|> construct instead, so the channel header is never recognized as reasoning and is surfaced verbatim as Text. Gemma4ToolCallAdapter.Parse already scrubs the sibling <|tool_response> / <tool_response|> turn markers (#150) but has no knowledge of <|channel> / <channel|>.

Suggested fix

Handle Gemma 4's channel markers in one of:

• Gemma4ToolCallAdapter — scrub <|channel>label<channel|> headers from PlainText (mirrors the existing <|tool_response> scrub), and expose them to the streaming state machine so the header isn't emitted as a content delta; or
• the engine's reasoning split — recognize Gemma 4's channel format so a thought/reasoning channel is routed to GenerateChunkKind.Thinking (and suppressed under enable_thinking=false), while a final/answer channel routes to Text.

In the observed output the content after <|channel>thought<channel|> was the actual final answer (no separate final channel was emitted), so at minimum the header markup must not reach Text; keep the content.

Impact / workaround

Downstream (Ayu, a voice assistant) the leaked thought label + angle-bracket markup would be read aloud. Worked around there for now with a stream-safe Ayu-side scrubber that strips <|channel>…<channel|> headers, but the natural home is the adapter/engine since it's Gemma 4 wire-format knowledge.

Repro sketch

1. Load Gemma 4 12B (arch gemma4), enable_thinking=false.
2. Render history: system + user + assistant(tool_calls:[read_file]) + {role:tool, content, tool_call_id} via Gemma4ToolCallAdapter.RenderToolResult.
3. add_generation_prompt=true, generate.
4. The first Text tokens are <|channel>thought<channel|> before the answer.

[pekkah]

Deeper root cause (from driving an in-process agentic loop)

The <|channel> leak is a symptom of a bigger problem: Gemma 4 12B does not stop after emitting a tool call — it keeps generating a hallucinated tool-response turn, and the channel header is part of that runaway continuation.

Raw model output for a single read_file tool turn (captured verbatim, enable_thinking=false, arch gemma4):

<|tool_call>call:read_file{path:<|"|>src/Ayu.Host/Program.cs<|"|>}<tool_call|><|tool_response>.<|channel>thought…

After <tool_call|> (the close marker) the model emits <|tool_response> (opening the tool-result turn itself), a stray ., then <|channel>thought<channel|> and proceeds to hallucinate the tool result and the final answer, running until MaxNewTokens (observed still generating at the 2048-token cap ~75s later). An agentic driver that wants to inject the real tool result and re-infer therefore:

1. pays a full wasted generation (up to MaxNewTokens) on the tool-call pass, and
2. has to suppress/scrub the hallucinated <|tool_response> … <|channel>thought<channel|> … tail.

Why generation doesn't stop

InferenceEngine (and ContinuousBatchingEngine) use sp.StopTokenIds ?? _tokenizer.EogTokenIds as the stop set. EogTokenIds is <end_of_turn>/<eos> — it does not include the tool-call close <tool_call|> nor the tool-response open <|tool_response>. A well-behaved model emits <end_of_turn> after its tool calls (→ stops); this Gemma 4 12B QAT emits <|tool_response> instead and runs on.

Confirmed working stop point

InferenceEngine.cs:995 (if (stopIds.Contains(next)) break;) breaks before emitting the sampled token (it's recorded in fullSeq for KV continuity only). So adding the <|tool_response> token id to the stop set:

• stops generation the instant the model finishes the tool call and tries to open the response turn,
• leaves raw == "<|tool_call>…<tool_call|>" — a complete, parseable block (the stop token isn't emitted), and
• eliminates the wasted tokens and the hallucinated <|channel> tail.

Verified downstream (Ayu) by setting SamplingParams.StopTokenIds = [..EogTokenIds, SpecialTokens["<|tool_response>"]] on tool-active passes.

Suggested upstream fixes

1. Expose a tool-boundary stop from the adapter. IToolCallAdapter could surface the token(s) that mean "the tool-call section is finished" (Gemma 4: <|tool_response>), so a host driving the engine in-process for agentic loops can stop cleanly without hardcoding model-family special tokens. Today a consumer must reach into tokenizer.SpecialTokens["<|tool_response>"] itself.
2. StopTokenIds is a replace, not an augment, footgun. sp.StopTokenIds ?? EogTokenIds means a consumer who sets StopTokenIds to add a tool-boundary stop silently loses the EOG set unless they remember to re-include it. Consider making StopTokenIds additive to EogTokenIds, or documenting the replace semantics on the property.
3. Channel markers (original report): <|channel>label<channel|> should be handled by the adapter/engine the way <|tool_response> already is (Server (OpenAI): tool-call response leaks <|tool_response> into content (should be null) + usage.prompt_tokens always 0 #150) — either scrubbed from text or, for a thought/reasoning channel, routed to GenerateChunkKind.Thinking and suppressed under enable_thinking=false. Note the post-tool generation prompt sometimes appears to prime <|channel>, so the model can emit a bare <channel|> close with no preceding open in the answer pass.