Feature request: image attachments with a text-only main model via a separate vision model (MCP vision-helper)

[Ryutaku]

Summary

When the active session model is text-only (e.g. deepseek-v4-flash, which the dsh-llm-deepseek adapter declares with inputModalities: ["text"]), the harness rejects image attachments at the GUI/host admission layer with:

当前模型不支持图片，请切换支持图片的模型 (MODEL_DOES_NOT_SUPPORT_IMAGES / attachment-error)

This blocks a very common private-deployment setup: a text-only main model + a separate multimodal model (e.g. Qwen3.6-27B served via llama.cpp) used as a "vision helper". The user never gets to attach the image — it is rejected before it reaches the session, so no agent-side routing to a vision tool is possible.

Use case

• Main model: text-only (DeepSeek official API).
• Vision helper: privately deployed multimodal model (OpenAI-compatible /v1/chat/completions, image_url with data URLs), already working as an MCP server exposing describe_image(path).
• Desired flow: user drags/pastes an image in the GUI → harness stores it as a durable attachment → the model receives a text placeholder with the attachment reference → the agent calls the vision MCP tool with that reference → the returned text description enters context → the text-only main model answers.

Current behavior (0.1.0-rc.6)

• Admission gate in dsh-host-apiproxy (session prompt admission and selectModel): rejects when modelInfo.inputModalities !== undefined && !inputModalities.includes("image").
• dsh-llm-deepseek hardcodes inputModalities: ["text"] in modelInfo() and in resolveModel()'s fallback branch.
• read_image (dsh-tool-fs) has a stricter gate: it requires the resolved route to declare image input explicitly (inputModalities === undefined is rejected), and it returns an image block — which the DeepSeek chat-completions adapter rejects with UNSUPPORTED_CONTENT, so it cannot serve as the vision bridge either.
• The MCP workaround works fine when given a file path (image stored anywhere readable), but the GUI attachment path never yields a reference the agent can see.

Feature suggestions (any of these would unblock the pattern)

1. Admission option for text-only models: a per-session/deployment setting that allows image intake even when the active model is text-only — persist the image as a durable attachment and give the model a text placeholder carrying the attachment id/reference (e.g. [图片 attachmentId=xxx]), so the agent can hand it to a vision tool. The provider adapter still never receives image bytes.
2. Modality override in model config: allow deployments to override inputModalities per provider/model (e.g. keep deepseek-v4-flash text-only on the wire, but let a deployment opt into image intake), instead of the adapter being the only source of truth.
3. Attachment reference exposure: when image admission is allowed for a text-only model, surface the sha256:/attachment reference to the agent (e.g. in the message text projection), and add a small host tool to resolve an attachment id to its stored path (or bytes), so MCP/vision tools can consume it without the user needing to manage files manually.

Notes on our current workaround

We built a stdio MCP server (FastMCP) wrapping the private multimodal endpoint, registered via cordis.patch.yml with @deepseek-ai/dsh-mcp-client (tools appear as mcp__vision__describe_image). It works end-to-end when invoked with a filesystem path. The only missing piece is the GUI attachment pipeline for text-only sessions.

Environment: dsh 0.1.0-rc.6, profile web, Windows host.

[shinjiyu]

This is the GUI admission gap in #588: text-only main model + a separate vision helper.

dsh-plugin-multimodal sits in the adapter layer, not as a vision_* toolbox.

• If the routed model already accepts images, attachments pass through unchanged.
• If it is text-only, the plugin advertises image input so the Web UI accepts the paste, then a vision sidecar describes the image and the main model only sees text.
• see_image covers screenshots already on disk.

It does not replace Anionex/dsh-vision-toolkit. That package is tools the agent must call. This one is so paste is not rejected.

Install:

dsh plugin --profile web add github:shinjiyu/dsh-plugin-multimodal

Then restart dsh. Set DSH_VISION_BASE_URL / DSH_VISION_API_KEY / DSH_VISION_MODEL to a real vision model (not deepseek-v4-flash / GLM-5.2-FP8).

Repo: https://github.com/shinjiyu/dsh-plugin-multimodal

[weijiafu14]

The missing GUI attachment pipeline in this exact setup is now implemented in pi2dsh@0.5.0, and it can reuse your existing mcp__vision__describe_image(path) server.

pi2dsh registers an explicit companion route only for models opted in through modelOverrides. The original deepseek-official route remains text-only and truthful; the companion is shown in the Web picker as DeepSeek + Vision Bridge and accepts pasted or dragged images.

For your existing MCP flow, the important behavior is the fallback path: the companion bridge materializes each admitted attachment to a temporary file and replaces the image block with an actionable text notice containing that path. The DeepSeek agent can pass the path directly to mcp__vision__describe_image. Before forwarding to the original adapter, the bridge recursively strips every image block, including nested tool-result content, so the official DeepSeek API receives text only.

The same bridge can alternatively run a converted Pi plugin such as @kassing/pi-vision for automatic VLM analysis and native context injection:

npx pi2dsh@0.5.0 convert @kassing/pi-vision --out vision-bundle
dsh plugin --profile web add file:$PWD/vision-bundle

Merge the companion opt-in into ~/.dsh/pi2dsh/agent/models.json:

{
  "providers": {
    "deepseek-official": {
      "modelOverrides": {
        "deepseek-v4-flash": { "input": ["text", "image"] }
      }
    }
  }
}

Restart dsh web, select DeepSeek + Vision Bridge, then paste the image normally. Your private MCP endpoint can remain exactly where it is; this supplies the attachment-to-path handoff that was previously missing.

We verified the real DSH Web loop with pasted image probes and also added contract coverage for path-bearing placeholders, nested image removal, original-route forwarding, and reporting the original text-only model identity to Pi plugins.

Full example and configuration:
https://github.com/weijiafu14/pi2dsh/tree/main/examples/vision-bridge