[Bug]: Response truncated due to output length limit — still occurring after #7237 fix (re-opening closed issue)

[aurelienp-alt]

Bug Description

The error "Response truncated due to output length limit" persists despite previous fixes in #7242 and #9525, which were merged in response to the now-closed issue #7237.

The issue was closed by a maintainer last month with the explanation that it is "not a bug" but rather a context window limitation. However, extensive testing and user reports (see the 10+ comments since the closure) continue to reproduce the error across different models — including those with very large context windows (414k tokens, 131k tokens, and even Gemini 1.5 with 2M tokens).

This suggests that while the input context might be sufficient, the output token cap or the continuation logic for long tool outputs/summaries is failing. This new issue is meant to re-surface the problem with specific reproduction data showing failures on fresh sessions with large-context models.

Steps to Reproduce

1. Launch hermes chat (CLI) or hermes gateway start (Telegram / Discord / Slack).
2. Trigger a tool call that returns a large, dense payload (e.g., >80KB).
   • Example: mcp_mongodb_collection_schema on a complex database collection.
   • Example: read_file on a large lockfile (e.g., pnpm-lock.yaml or package-lock.json).
3. Observe the agent stop mid-generation (often during a summary or a sub-agent's response) with:
   :warning: Response truncated (finish_reason='length') - model hit max output tokens
:warning: Truncated tool call response detected — refusing to execute incomplete tool arguments.
Error: Response truncated due to output length limit
4. This happens even on fresh sessions (low input context), ruling out simple history overflow.

Expected Behavior

Long responses should be split into chunks or the continuation logic should reliably retry until a complete response is assembled. The 3-retry mechanism introduced in #7242 / #9525 should prevent hard truncation in all API modes (chat_completions and anthropic_messages) by detecting finish_reason='length' and requesting the remainder.

Actual Behavior

The agent throws "Response truncated due to output length limit" and halts. The continuation/retry logic does not trigger effectively, or the truncated JSON/summary is rejected by the agent's core before it can be completed.

Confirmed still occurring in:

• qwen3.6-27b-fp8 (414k context) — @theogravity
• GLM-51-NVFP4 — @Damikk19
• gpt-oss-20b (131k context) — @kevinctofel
• aws-claude-opus-4-6 — @yangyang233333
• Gemini 1.5 Flash / Pro (2M context) — @poulpe-agent-repro
• custom provider: gpt-5.5 — @arsitekberotok

Affected Component

Agent Core (conversation loop, tool output handling, continuation logic)

Debug Report

Report       <https://paste.rs/JFt5G>
agent.log    <https://paste.rs/Anyts>
gateway.log  <https://paste.rs/N8CNm>

Operating System

Ubuntu 24.04

Root Cause Analysis

The current logic fails to differentiate between "Context Window Full" (Input) and "Max Output Tokens Hit" (Output).

• Many providers have a hard limit on single completion length (e.g., 4096 or 8192 tokens).
• When a tool returns a massive result (e.g., a 100KB schema), the model tries to process/summarize it and hits its output cap long before it exhausts its input context.
• The continuation loop's progressive max_tokens boost (capped at 32k per fix(agent): increase tool call truncation retries from 1 to 3 with max_tokens boost #9525) might be insufficient or is not correctly detecting truncation when it happens inside a tool-call or sub-agent structure.

Proposed Fix

• Intelligent Tool Output Management: If a tool result exceeds a certain size (e.g., 50KB), save it to a temporary file and only inject a summary/snippet into the context to prevent the model from "choking" on the output generation.
• Unify Continuation Logic: Ensure anthropic_messages and chat_completions paths are perfectly synced regarding truncation detection and retry.
• Configurable Output Caps: Allow users to override the 32k boost cap or the default max_tokens for completions in config.yaml.
• Improved Logging: Surface the actual token counts (Output used / Max output) when a truncation occurs to distinguish from Input context limits.

Are you willing to submit a PR for this?

• I'd like to fix this myself and submit a PR

[daimon-nous]

Related prior work:

• [Bug]: Error: Response truncated due to output length limit #7237 (original issue, closed) / fix(agent): add length-continuation support for anthropic_messages mode #7242 (anthropic_messages continuation fix) / fix(agent): increase tool call truncation retries from 1 to 3 with max_tokens boost #9525 (retry boost PR)
• fix(agent): boost max_tokens on length-continuation retries #9489 (progressive max_tokens boost PR, open)
• [Bug]: NVIDIA Build API modle z-ai/glm4.7 returns model hit max output tokens #9372 (NVIDIA model truncation, open, same symptom)
• fix(agent): preserve continuation output cap after truncation #26427 (fix PR targeting this issue)

[teknium1]

Fixed in PR #36705 (merge commit 023149f on main).

Root cause: a stream that drops mid-response after tokens are delivered (peer-closed connection / stale-stream reconnect) is converted into a synthetic finish_reason="length" stub. The conversation loop treated that network stall as an output-cap truncation — when the dropped content was a tool call it retried only once, then hard-failed with "Response truncated due to output length limit," even on large-output models that never hit any cap (e.g. Opus).

The fix:

• Tool-call truncation now retries up to 3× (was 1) with a progressive max_tokens boost, and is stub-aware — a network stall reports "Stream interrupted mid tool-call — retrying (n/3)" instead of the false "model hit max output tokens," and both the give-up message and returned error distinguish a network drop from a real truncation.
• Length-continuation retries preserve the original request's output cap as a floor, so a high provider/model default isn't silently downshifted to 8K/12K.

Salvaged the substance of #26427 and #9525, adapted to the post-refactor conversation_loop.py. Credit to @LeonSGP43 and @ygd58 (Co-authored-by). Thanks for the detailed re-open with reproduction data.