[Bug] Grok gateway leading <think> content is rendered as answer text (opt-in parser proposal)

[Rotor1996]

Grok gateway <think> output stays in the answer instead of the reasoning disclosure

I am reporting this through Discussions because CONTRIBUTING.md says external PRs are not currently accepted. I have a tested, opt-in compatibility patch available for reference; this is not a request to bypass that policy.

Observed behavior

With @deepseek-ai/dsh-llm-pi-ai and an OpenAI Chat Completions-compatible gateway serving grok-4.7 / Low, some responses contain a leading <think>...</think> section in delta.content. Harness displays those tags and their contents in the ordinary answer, while its reasoning disclosure remains separate.

The same route can return native reasoning_content in another request. A tool round can also end the text block before </think> arrives. Therefore this report is about a gateway's mixed output representations; I have not established whether the gateway or its upstream model introduces the tags, and I am not claiming that the direct xAI API behaves this way.

Checked against official master at 86a402200d95fecef4529d969836f1fe13c6cb0d (0.2.1-alpha.4, pi-ai 1.1.0) on Windows / Node 24.

Minimal reproduction

A deterministic Chat Completions stream can send the following payloads (each as an SSE data: record):

{"choices":[{"index":0,"delta":{"role":"assistant","content":"<thi"},"finish_reason":null}]}
{"choices":[{"index":0,"delta":{"content":"nk>checking the result</think>42"},"finish_reason":null}]}
{"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

Follow with data: [DONE].

Current result: one text block containing <think>checking the result</think>42.

With explicit compatibility enabled: a reasoning block containing checking the result, followed by a text block containing 42; the existing UI can display the reasoning in its disclosure. Ordinary prose that quotes tags should remain unchanged.

Proposed compatibility option

Independent fix branch · single commit · diff against the checked upstream base.

The patch adds parseThinkTags: true to a model entry or modelOverrides. It is disabled by default and accepted only for openai-completions models. For example, inside an existing gateway route:

models:
  - id: grok-4.7
    parseThinkTags: true

The parser:

• Recognizes only an exact leading <think> section in a native text block, including tags split across chunks. An unclosed section remains reasoning until that block ends.
• Leaves tags embedded in ordinary prose/code fences, <thinking> tags, and later tags after the first closing section literal.
• Preserves native reasoning, tool calls, usage and response identity, while remapping block indexes and aligning Chat Completions history replay metadata.
• Buffers interleaved blocks only until the initial text block has nonempty output; subsequent blocks follow their emitted order.
• Measures idle activity before normalization and preserves the current adapter's cancellation/cleanup behavior.
• Reports EMPTY_RESPONSE if stripping empty tags leaves an otherwise successfully completed response with no content; tool-only, truncated, aborted and failed responses retain their existing outcomes.

This changes the provider adapter/configuration and adds regression coverage and documentation. It does not modify the Agent loop or the UI/model selector, and does not rewrite existing conversations.

Validation

On the branch above:

• 203 focused adapter, conversion, parser, idle-watchdog and real-Loader composition tests passed.
• 65 focused parser/Loader/watchdog tests passed with 100% statement, branch, function and line coverage for think-tags.ts.
• A real Grok two-round tool/history smoke passed. Reasoning is optional in that smoke because a valid response may omit it; deterministic SSE/Loader tests cover tag splitting independently.
• The keyless Web scenario passed, including nonempty reasoning, a tool round, expand/collapse, reload and persisted Session replay. Its fixture was refreshed through the repository workflow for current runtime-context records.
• pnpm run build passed all 9 build/check steps; pnpm run lint passed; commit hooks passed.
• pnpm run doc-sync: 42 passed, 1 failed. The failure is in unchanged upstream python/sdk/uv.lock:113: the wheel size 6344383 happens to resolve to a commit prefix in my local object database, so verify-repository-references flags it. The lockfile is identical to the checked upstream base; no unrelated workaround is included.

The real gateway endpoint and credentials are intentionally omitted. The branch includes a key-gated think-tags.e2e.ts smoke using DSH_GROK_BASE_URL, DSH_GROK_MODEL and DSH_GROK_API_KEY, plus keyless regression tests.

Would an explicit per-model compatibility option like this fit the adapter's intended scope, or would you prefer this normalization to live in an external middleware plugin?

[argszero]

The compat table already models the opposite direction — that asymmetry is the useful framing

Your reading is sound, and there is prior art in the tree that names this precisely.

The knob you want exists for the other direction. The pi-ai compat table carries (packages/llm/llm-pi-ai/src/catalog.ts:411-412):

/** Whether thinking blocks must travel as text in `<thinking>` delimiters; `openai-completions`. */
requiresThinkingAsText?: boolean

That models outbound: "this endpoint wants thinking text wrapped in delimiters." Your gateway does the inbound thing — it sends <think>…</think> inside delta.content — and there is no counterpart flag, which is why nothing splits it. So the option you propose is not a new concept for this table; it is the missing twin of one that already exists, and saying so is probably the strongest argument for it.

Where the tags arrive, and where a fix can stand. With llm-pi-ai, content deltas map one-for-one (packages/llm/llm-pi-ai/src/stream.ts:157-163):

case 'text_delta':
  yield { type: 'text-delta', index: event.contentIndex, text: event.delta }
case 'text_end':
  yield { type: 'block-end', index: event.contentIndex, block: { type: 'text', text: event.content } }

Your <thi / nk>… split across two SSE records is therefore two text-delta chunks at the same index, so the split needs cross-chunk state — and because text_end carries the whole accumulated block.text, any parser that rewrites deltas must rewrite the block-end payload too, or the finished block will contradict the stream.

The response side is a legitimate place for this and needs no core change. llm/stream is a waterfall whose next() returns the chunk iterable (packages/llm/llm/src/index.ts:75, dispatched at :1176-1179), so a plugin can wrap the stream and restructure it. That is the right home for an opt-in parser: mounted by deployments that have such a gateway, absent everywhere else — which matches your framing exactly, and means your patch does not have to be right for everyone to be useful.

Two things it has to get right, both from your own report:

• Block indices must be renumbered when one text block becomes a reasoning block plus a text block, or the stream's block identity breaks downstream.
• The tag may never close inside a turn. You noted a tool round can end the text block before </think> arrives; a parser that buffers without a flush path moves the failure from "tags visible" to "answer swallowed", which is worse. Your requirement that ordinary prose quoting the tags stays unchanged also rules out a naive global strip.

One caveat on reachability. A response-side wrapper can restructure the chunk stream, but it cannot reach anything an adapter has already folded into a finished block before yielding it — I hit exactly that limit on a different marker in this same seam family, where the provider accumulated the sentinel internally and the terminal event carried no chunk to intercept. If you ever see tags survive despite a correct parser, check the adapter's block-end payload path before reworking the parser.

I cannot reach your gateway from here, so I have not re-run your reproduction against a live endpoint; the minimal SSE records and your patch branch are what make the report actionable.