【BUG】read_image 图片块在 openai-responses 协议下未做能力适配，触发 400 且会话永久卡死

[lingxinsuixing]

1. 摘要（DSH 侧问题）

DSH 中配置为 pi-ai openai-responses 协议的 provider（本例为 agnes / agnes-3.0-flash），在 read_image 工具结果含本地图片块 {"type":"image","attachment":{...}} 后，该图片块被原样拼入下一次请求的 input，网关拒绝整条请求（400 json_parse_error，input 反序列化失败）。该图片块停留在会话历史里，导致该会话之后所有请求（含纯文本）全部 400，会话永久卡死、无恢复路径，用户只能新建会话。

DSH 的 dsh-llm 已具备图片降级原语（textOnlyImageText、projectImagesForTextModel、requiredImageOffload 等），但 pi-ai 的 openai-responses 路径并未在组装请求时按 provider/协议的图片输入能力对图片块做降级或剔除，也未对这类 400 提供会话自愈。

2. 环境

• Windows，DSH Web GUI（dsh-primary-runtime 0.2.1-rc.2）
• provider：agnes（https://api.agnes-ai.cn/v1），模型 agnes-3.0-flash，协议 pi-ai openai-responses
• 出事故会话：session-3188ae7b-ea39-460a-a6c5-098310da31f6
• 该模型本身支持图片输入（视觉模型），但其 Responses 端点对图片块的接受情况与文档描述不符（见 §4.1），属于 provider 侧环境事实，不作为本 bug 的归因主体

3. 复现步骤（DSH 内操作）

1. 在协议为 openai-responses 的 provider 会话中，让 agent 调用 read_image 读任意本地图片
2. 工具结果返回后（content 含 {"type":"image"} 块）
3. 下一条 LLM 请求 → 400
4. 之后任意用户消息（纯文本）→ 全部 400，会话卡死

4. 证据

出事前该会话已正常执行 30+ 次 LLM 请求（56K token 历史、含 reasoning/thinking 回放），全部成功；第一次 400 恰好出现在 read_image 图片块首次进入历史那一步，唯一新变量即该图片块。

涉事工具结果内容：

[
  {"type": "text", "text": "<path>…010.webp</path>\n<type>image</type>\nimage/webp image, 512x512 px, 33116 bytes"},
  {"type": "image", "attachment": {"attachmentId": "sha256:3ffcabdc…", "mediaType": "image/webp", "bytes": 33116, "width": 512, "height": 512, "name": "010.webp"}}
]

DSH 报出的三次 400 原文（网关侧 input 反序列化失败）：

agnes API error (400): {"message":"***.BadRequestError: OpenAIException - {\"error\":{\"message\":\"Invalid JSON data: Failed to deserialize the JSON body into the target type: input: data did not match any variant of untagged enum ResponseInput at line 1 column 422733\",\"type\":\"invalid_request_error\",\"code\":\"json_parse_error\"}}"}

• 三次失败列号 422557 → 422664 → 422733，随会话尾部追加内容单调右移，失败点位于请求体尾部（图片块所在区域）
• line 1 表明请求体为单行 JSON，body ≥ 422KB

4.1 网关侧实测（环境佐证，2026-10-28，curl + 有效 key）

请求	结果
POST /v1/chat/completions 纯文本	200
POST /v1/chat/completions + 公开图片 URL	200，image_tokens: 176
POST /v1/chat/completions + base64 data URL	200，image_tokens: 176
POST /v1/responses 纯文本（连试 3 次）	全部 500 bad_response_body
POST /v1/responses + 图片块（3 种形态）	400，图片 part 被标 type:'unknown'
GET /v1/models	200

即该 provider 的 Responses 端点当前对图片块不可用，Chat Completions 端点完整可用。此事实仅用于说明「该协议路径下图片块没有合法形态」，不是本 bug 的归因主体；DSH 侧的问题在于：即便目标 provider 的某协议不支持图片块，也应在请求组装时降级/剔除图片块（或按 provider 声明的可用协议发送），而不是让图片块卡死整个会话。

5. 期望行为（DSH 侧）

• 组装请求时按 provider + 协议的图片输入能力处理图片块：不支持时降级为文本占位（[image: 010.webp 512x512 33116B]，复用 dsh-llm 已有原语），支持时以该协议接受的形态发送
• 捕获此类 400（请求体校验失败）后提供会话自愈：清理历史中的图片块后自动重试，或明确提示「本会话无法恢复，请新建会话」

6. 用户侧规避（配置层面，非 DSH 缺陷）

本例的 agnes provider 属个人配置问题：桌面端把 api 声明为 openai-responses（该端点对图片块不可用），而 Chat Completions 端点完整可用。规避方式（无需等 DSH 修复）：

• 将该 provider 的 api 改为 openai-completions（chat 端点文本与图片均可用，本地 base64 图片 200）
• 或将该模型的能力声明降为纯文本（input: [text]），避免 DSH 走图片输入路径

[PerryLink]

这里的判据不是协议，是模型有没有声明图片输入能力。

packages/llm/llm/src/index.ts:1105-1109：

if (modelInfo.inputModalities !== undefined
  && !modelInfo.inputModalities.includes('image')
  && projectedMessages.some(message => contentHasImage(message.content))) {
  projectedMessages = projectImagesForTextModel(projectedMessages)
}

只有模型明确声明「不包含 image」时才降级成文字形式。inputModalities 来自 pi-ai 模型的 input 数组（llm-pi-ai/src/adapter.ts:309、:336）。你的 agnes 是视觉模型，input 里大概有 image ⇒ 走不到这条降级，图片块原样进请求，收不收由网关决定。

所以「openai-responses 未做能力适配」这个说法要改一下：适配是按模型能力做的，不按协议。想让 DSH 降级，就在模型声明里去掉 image——这样请求会带上 textOnlyImageText 的文本形式，而不是图片块。

会话卡死那半是另一回事：400 之后那条图片块还留在历史里，之后每条请求都会重复它，这条没有自愈路径。

我只读了源码（f3803c96），没有跑你的 provider。

[argszero]

PerryLink's reading above is correct, and I re-measured it on a newer line — so this is a confirmation, then the part that came from running it rather than reading it.

Confirmed at 0.2.1-rc.3 (the report is on rc.2; the tree read above was f3803c96). The predicate at packages/llm/llm/src/index.ts:1105-1109 is exactly as quoted, and textOnlyImageText (packages/llm/llm/src/content.ts:76) plus projectImagesForTextModel (:353) are still the primitives behind it. Measured with a recording adapter: with the model declaring ['text','image'], the image block is what the adapter is asked to send; with ['text'], core substitutes the placeholder itself. That negative control is what puts the defect in the condition and not in the degradation — the degradation already works, it just is not reached.

One place I would push back. "Remove image from the model declaration" is the report's workaround 2, and it is global: the same model id, served by a route that can carry images, loses them there too. The reporter's §4.1 is what makes that concrete — one provider, two protocols, /v1/chat/completions accepts the same image by URL and by base64 while /v1/responses fails on pure text and marks image parts type:'unknown'. Capability is declared per model; carriage is a property of the route. That mismatch is the gap, and it cannot be expressed by editing inputModalities.

The second half — the block stays in durable history and is replayed into every later request, including pure-text turns and every compaction of the same prefix — has no recovery path, which I agree with.

What I published, so nobody has to choose between losing images everywhere and losing the session. @argszero/dsh-image-route-guard v0.1.0:

dsh plugin add github:argszero/dsh-image-route-guard#v0.1.0

- id: image-route-guard
  config:
    routes:
      - provider: agnes
        model: agnes-3.0-flash   # omit to cover every model on that provider

It projects image blocks to the harness's own placeholder before dispatch, only on the routes you declare image-incapable. It reproduces core's textOnlyImageText output byte-for-byte (pinned by a test against the runtime's own export), and it walks tool results as well as user messages — which is the reported shape, since the image came back from read_image. A route you did not name, a call with no image, and a request it cannot read are dispatched with the caller's own options object, not a copy, and silently.

The honest limit. This removes the poisoned block from the request. It does not teach the provider path to degrade images, and it cannot detect by itself which protocols can carry one — an undeclared route behaves exactly as before. The real fix is the predicate above consulting the route; when it does, the mount becomes unnecessary.

Measured on @deepseek-ai/dsh-llm@0.2.1-rc.3 with @deepseek-ai/cordis@4.0.5. 17 arms: five positives, five negatives (undeclared provider, other model on a pinned rule, image-free call, empty config, teardown restore), two fail-open, plus packaging pins. Installing it is verified the hard way as well — a clean DSH_HOME, the real dsh plugin add verb, the entry dsh inserts in --dump-config, and the installed copy imported and mounted.

If a session is hit by an image the route rejects at runtime rather than one it has no wire shape for, @argszero/cordis-plugin-image-offload-fallback covers that case; it acts after the failure, this one acts before the request.

[PerryLink]

You're right, and the gap you name is real. The predicate cannot express "this route cannot carry an image", so my "drop image from the model declaration" was too blunt — it would also turn off images on routes that do carry them.

Worth pinning where the information actually lives, because it decides how small the real fix is. The predicate reads modelInfo (packages/llm/llm/src/index.ts:1105), which is the resolved model's info; the protocol is the adapter's own fact — llm-pi-ai/src/catalog.ts:341 maps openai-responses to its compat gate, and the model carries api. So the missing input exists one layer down: the adapter could carry a per-route "carries images" flag into the model info, and the same projection would then run for exactly the routes you declare — no plugin needed.

Until that exists your plugin is the honest workaround, and I'd put your sentence in the report title: declared per model, carried per route.

Source read at f3803c96; I have not run your arms.