Detect and recover from degenerate repeated model output (repetition guard) / 检测并恢复退化的重复模型输出（重复守卫）

[filantropo17]

Problem / 问题

Models sometimes degenerate into repeating one word, phrase, or n-gram until the output-token ceiling. The harness currently accepts such responses as an ordinary completed turn: the stream finishes with stop or max-tokens, neither of which is a failure, so nothing truncates the loop and no retry is triggered. The user is left with a long, unusable, token-wasting reply, and the session log is polluted with the repeated content.

模型有时会退化为反复输出同一个单词、短语或 n-gram，直到打满输出 token 上限。当前 harness 会把这类响应当作普通的正常完成 turn：流以 stop 或 max-tokens 结束，两者都不算失败，因此没有任何机制截断循环或触发重试。用户只能得到一段冗长、不可用且浪费 token 的回复，会话日志也被重复内容污染。

Proposal / 提议

A stream-level repetition guard: wrap the existing llm/stream waterfall, detect when the trailing text is one unit repeated N times, truncate the stream, and fail the request with a new provider-neutral code DEGENERATE_OUTPUT, so the existing agent/request-error → dsh-llm-retry path retries it under the provider policy. No agent-loop changes are needed — the llm/stream waterfall already sits at exactly the right seam.

流级重复守卫：包装现成的 llm/stream 瀑布，当尾部文本出现同一单元连续重复 N 次时截断流，并以新的 provider 无关错误码 DEGENERATE_OUTPUT 判失败，从而让现有的 agent/request-error → dsh-llm-retry 路径按 provider 策略重试。完全不需要改 agent 主循环——llm/stream 瀑布正好位于正确的接缝上。

I built this as an ecosystem plugin (dsh-llm-repetition-guard) to prove the approach:

我把它实现为生态插件（dsh-llm-repetition-guard）来验证该方案：

• One detector per stream; per-block text accumulation under a bounded sliding window (default 1024 chars).
  每个流一个独立检测器；按块在受限滑动窗口（默认 1024 字符）内累计文本。
• Detection is the longest trailing run of one unit over every alignment — "data data data …" is caught whether the last unit lands on a window boundary or not.
  检测依据是某单元在所有对齐方式下的最长尾部连续出现——"data data data …" 无论最后一个单元是否恰好落在窗口边界上都能被捕获。
• text deltas are guarded by default; reasoning deltas are ignored unless opted in (reasoning text is pattern-heavy and would trip conservative thresholds); tool-call argument streams are never guarded (JSON legitimately repeats keys and separators).
  默认只检测 text delta；reasoning delta 默认忽略（推理文本天然模式化，保守阈值容易误伤）；工具调用参数流永不检测（JSON 合法地重复键与分隔符）。
• On detection it yields a terminal DEGENERATE_OUTPUT error finish and truncates the stream; the loop retries under the provider retry policy, and truncated chunks never enter derived messages.
  检测到时产出终止性的 DEGENERATE_OUTPUT error finish 并截断流；主循环按 provider 重试策略重试，被截断的 chunk 永不进入派生消息。
• Conservative defaults: start detecting after 64 accumulated chars, trip at ≥6 consecutive repeats of an 8-char unit. All thresholds are configurable.
  保守默认值：累计 64 个字符后开始检测，同一 8 字符单元连续出现 ≥6 次即触发。所有阈值均可配置。

What I'd like feedback on / 希望得到反馈

1. Default retryableCodes. Should DEGENERATE_OUTPUT join the default retryable set in dsh-llm? The published package currently omits it, so deployments must opt in explicitly. Both EMPTY_RESPONSE and DEGENERATE_OUTPUT are content-quality failures that produced no usable output, so repeating them is safe.
   默认 retryableCodes。 DEGENERATE_OUTPUT 是否应该加入 dsh-llm 的默认可重试集合？当前已发布的包未包含它，部署方必须显式配置。EMPTY_RESPONSE 与 DEGENERATE_OUTPUT 都是未产生可用输出的内容质量类失败，重复它们都是安全的。
2. Core vs. ecosystem. Should the harness ship a content-quality extension point in core (the llm/stream wrapper pattern) rather than relying on ecosystem plugins for this class of checks?
   核心 vs. 生态。 harness 是否应该在核心中提供内容质量扩展点（llm/stream 包装模式），而不是依赖生态插件来承载这类检查？
3. Thresholds. Is "≥6 consecutive repeats of an 8-char unit after 64 chars" a reasonable conservative default? Any false-positive experience on real workloads?
   阈值。 "累计 64 字符后，8 字符单元连续重复 ≥6 次"作为保守默认值是否合理？真实负载下是否有误报经验？
4. Beyond consecutive repetition. Alternating phrases or sparsely scattered short units are not detected today — is that a class worth covering?
   非连续重复。 交替短语或稀疏散布短单元的输出目前无法检测——这一类是否值得覆盖？

The plugin is published as dsh-llm-repetition-guard and carries an ./invariant companion that verifies the truncation contract (nothing follows a DEGENERATE_OUTPUT finish within one attempt).

插件以 dsh-llm-repetition-guard 发布，并附带 ./invariant 伴随插件，用于验证截断契约（一次 attempt 内 DEGENERATE_OUTPUT finish 之后不得再有内容）。

[denial123789]

The llm/stream waterfall looks like the right provider-neutral detection seam, but I would keep DEGENERATE_OUTPUT out of the default retry set initially. Detection and recovery have different risk: a false positive rejects valid output, while a retry creates another billable provider attempt.

A safe first contract would be:

• guard text deltas only; ignore reasoning by default and never scan streamed tool-call arguments with a text heuristic;
• keep a bounded trailing window and test across arbitrary chunk boundaries;
• terminate once with a stable LlmFailure code and yield nothing after terminal finish;
• require explicit normal-mode opt-in with a finite retry budget; direct ctx.llm.stream() consumers remain single-attempt;
• measure precision/recall on multilingual prose, code, tables, logs, Unicode, and legitimate max-tokens completions before promoting any thresholds to defaults;
• define live-client behavior because failed chunks are not durably committed, but text already painted by a UI is not automatically retracted.

The proposed 64-character / six repeats / eight-character unit is a useful candidate configuration, not yet a universal default. I wrote up the rc.8 stream and retry boundaries, an evaluation matrix, config example, UI semantics, and acceptance gates here: https://sandbaseai.github.io/deepseek-harness-handbook/degenerate-model-output.html