fix(gateway): keep reasoning-model conversations from stalling#46
Merged
Conversation
An empty turn (clean finish, no assistant text) is anomalous; treat it like an error for diagnosis and dump the request/response/finishReason/usage to the console before showing the generic fallback bubble. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add getUpstreamExchange tool to the evaluator's toolset to inspect raw upstream request/response pairs for physical-request entries. Mirrors the existing getTraceEntry tool pattern using pickRecorder/compare-mode design. Handles truncation notes and missing upstream gracefully. Includes unit tests verifying response formatting and null-handling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use optional-chaining on v-if guard; expand single-line div to clear singleline-html-element-content-newline lint warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix stale references flagged in final review: capture lives in
api/src/models/{capturing-fetch,operations}.ts and the evaluator tool in
ui/src/traces/evaluator-tools.ts; rawChars is a character count, not bytes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scoped via DEBUG=agents:downstream:<type>:<id>; serialisation only runs when the namespace is enabled, independent of trace storage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
# Conflicts: # api/src/models/operations.ts
A reasoning ("thinking") moderator spends the whole maxOutputTokens:100 budget
on reasoning_content, returns finish_reason:"length" with content:null, and the
generateObject parse rejects -> the gate fails open (every message allowed). It
also can't beat the 3.5s gate. Document this at the call site and expand the
moderator role description in the settings form (EN+FR) so admins are warned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eaming Scaleway's glm-5.2 deployment returns finish_reason:"stop" with no tool-call deltas under stream:true, while the identical stream:false request returns the tool call. The app streams for UX, so tool-using turns (e.g. the evaluator) collapsed to a "narrate then stop" preamble and looked interrupted. Proven by toggling only `stream` against the endpoint; isolated to Scaleway (not LiteLLM) and to glm-5.2 specifically (qwen3.5-397b, qwen3-235b, gpt-oss-120b, devstral all stream tool calls fine). streamedToolCallsBroken() flags scaleway|openai-compatible + /glm/; when set and tools are present the gateway runs the upstream call non-streaming (generateText) and re-emits the SSE chunks through the existing moderation-gate machinery, so the client still gets a normal stream. Removable once Scaleway fixes it; recheck with dev/scripts/scw-toolcall-probe.mjs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ai-sdk/openai silently drops the OpenAI-compatible `reasoning_content` field, so
reasoning models (GLM, DeepSeek, Qwen-thinking, …) had their thinking discarded at
the provider boundary. Route reasoning-capable OpenAI-compatible endpoints through
@ai-sdk/openai-compatible (the ai-v6 tag, 2.0.52) which maps reasoning_content to
reasoning parts, and carry it end to end:
- api: scaleway and openai-compatible('compatible') use createOpenAICompatible.
- gateway: forward reasoning-delta on the OpenAI `reasoning_content` SSE channel
(streaming, the non-streaming-upstream GLM workaround, and the non-stream branch).
- ui: the client provider is openai-compatible too (else it would drop reasoning on
the gateway→client hop); agent-stream-parts accumulates reasoning onto the message.
- ui: render reasoning as a collapsed "thinking" panel above the answer.
Verified end to end against the real Scaleway/LiteLLM endpoint (reasoning captured in
both modes) and with a mock reasoning seam ("reason") + gateway tests. Confirmed
reasoning round-trip across tool steps is NOT required for GLM (no 400, coherent
follow-up), so history does not echo reasoning back.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The moderator runs on a tiny token budget (maxOutputTokens:100) behind a 3.5s
gate. A reasoning ("thinking") model spent the whole budget on reasoning_content
and returned content:null, so the verdict parse failed and moderation fell open
(every message allowed) — verified against Scaleway glm-5.2.
Send reasoning_effort:none on the moderator call (and the admin probe), which
Scaleway/GLM honour: thinking is disabled, the JSON verdict comes back in ~8
tokens, well within budget and the gate. Verified end to end: without it,
generateObject throws "no object generated"; with it, {"action":"allow"} in 7
tokens. It is keyed by the shared OPENAI_COMPATIBLE_PROVIDER_NAME (scaleway +
openai-compatible now use one provider name) and ignored by other providers;
non-reasoning OpenAI-compatible models reject reasoning_effort with a 400, so
those retry once without it (isReasoningEffortRejected).
Requires the proxy to pass reasoning_effort through: see the litellm
allowed_openai_params change for glm-5.2-scw (separate infra repo).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
History compaction doesn't need to think. Send reasoning_effort:none for the summarizer role so reasoning models produce the summary fast and cheap (same rationale as the moderator). Ignored by non-openai-compatible providers; a non-reasoning OpenAI-compatible model that 400s on the param falls back to a plain call (isReasoningEffortRejected). Requires the proxy to pass reasoning_effort through (litellm allowed_openai_params, separate infra repo). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reasoning models' "thinking" was always surfaced as a foldable panel above each answer, which adds noticeable verbosity. Add a per-user "Full reasoning display" flag (default off) that switches between two modes: - Compact (new default): no reasoning panel; the transient "Thinking…" activity line is the only feedback and nothing persists afterward. - Full: the previous collapsed "Reasoning" panel. Reasoning is still always captured onto the message, so the flag is purely render-time: toggling it never resets the conversation and is reversible mid-chat. The flag lives in the existing service-scoped agent-chat-flags cookie next to toolExploration/subAgents/mermaid, toggled from the Settings tab. EvaluatorChat passes show-reasoning=true (admin tool wants full visibility). Server (api/src/traces) is untouched: render-only flag, its cookie parser ignores unknown fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the optional `upstream` field that recorded the raw gateway→provider exchange in every consented trace document, along with its capturing-fetch wrapper, evaluator getUpstreamExchange tool, trace-viewer panel, and test suite. The same raw bytes remain available on demand via the per-provider DEBUG=agents:upstream:* logging, which has no storage cost or per-request overhead when disabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ctation reconstructTrace falls back to the UI DEFAULT_FLAGS when no request carries flags; that default gained showReasoning, so the unit expectation needed the new key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
albanm
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Harden the gateway against conversations that stall or silently drop when the
model is a reasoning model on a Scaleway / OpenAI-compatible provider.
mode) through
@ai-sdk/openai-compatiblesoreasoning_contentis parsed insteadof dropped; forward it on the
reasoning_contentSSE channel and show it in anopt-in, render-only "Reasoning" panel (off by default, never resets the chat).
glm-5.2drops tool calls when streaming butreturns them non-streamed;
streamedToolCallsBroken()runs the upstream callnon-streaming when tools are present and re-emits the normal SSE stream. Includes
dev/scripts/scw-toolcall-probe.mjsto detect when the workaround can be removed.budget on hidden reasoning and returned
content:null→ fail-open. Sendreasoning_effort:none, retrying once without it when a non-reasoning model rejectsit with a 400.
DEBUG=agents:upstream:*/agents:downstream:*logging (never logs headers), and a console dump of the physical request/response
on an empty assistant turn.
Why: reasoning models on Scaleway/OpenAI-compatible providers made conversations
look interrupted — the assistant would "narrate then stop", moderation failed open,
and reasoning was discarded.
Heads-up:
streamedToolCallsBrokenforces non-streaming for any openai-compatibleprovider serving a
glmmodel, not only Scaleway — a native GLM endpoint that streamstool calls fine would lose incremental streaming (UX only, not correctness).