Model settings UI never exposes the catalog schema existing inputModalities — image-capable custom models are silently rejected

[Dxboy266]

Problem

Custom OpenAI-compatible models that actually accept images (served through gateways) are rejected at send time with the generic toast "The current model does not support images". The model editor in Settings - Models offers no way to fix this, even though the server side already supports it.

Root cause

Everything needed is already in place - only the settings UI does not expose it:

• packages/llm/llm-deepseek/src/index.ts - the catalog schema accepts inputModalities, but defaults to ["text"], so an unconfigured model is explicitly text-only.
• packages/api/session-controller/src/commands.ts - the send gate rejects an image when inputModalities is defined without "image" (MODEL_DOES_NOT_SUPPORT_IMAGES).
• packages/client/ui-settings-models/src/client/DeepSeekModelsEditor.tsx - each row only writes id / name / contextWindow / maxTokens.

So a user whose gateway model supports images has no in-product way to declare that, and every image send fails with copy that only says "switch models".

Suggested change

Expose an image input checkbox in each model row disclosure: checked writes ["text", "image"], unchecked removes the key (adapter default). Also point the rejection copy at that setting instead of only suggesting a model switch.

I have this working on my fork (7 files, includes zh/en copy and test updates):

Diff: Dxboy266/deepseek-harness@master...feat/model-image-input-toggle

Happy to adjust naming/UX to whatever direction the team prefers - understood review capacity is limited, so posting here per CONTRIBUTING pointer to Discussions.

中文摘要

自定义 OpenAI 兼容模型实际支持图片，但 llm-deepseek 目录 schema 的 inputModalities 默认 ["text"]，且 Web 设置的模型编辑器没有暴露该字段，导致发图固定被 MODEL_DOES_NOT_SUPPORT_IMAGES 拒绝。建议在模型行的展开区加图片输入开关（写入/删除 inputModalities），并把拒绝文案指向该设置。已在 fork 实现并附 diff。

[argszero]

Verified all three claims against 4e84901e64 (0.1.2-alpha.4) — they check out exactly, and there is one semantic nuance that makes your fix shape the right one.

Claim-by-claim confirmation:

1. packages/llm/llm-deepseek/src/index.ts:172 — inputModalities: z.array(z.union(MODEL_MODALITIES)).min(1).default(['text']). Every catalog entry resolves with inputModalities defined; an omitted key becomes an explicit ['text'], never "absent".
2. packages/api/session-controller/src/commands.ts:319-323 — the gate is if (model.inputModalities !== undefined && !model.inputModalities.includes('image')) → MODEL_DOES_NOT_SUPPORT_IMAGES. So rejection happens precisely when the field is defined without image.
3. packages/client/ui-settings-models/src/client/DeepSeekModelsEditor.tsx:20 — type CatalogField = 'id' | 'name' | 'contextWindow' | 'maxTokens'; the editor writes only those four.

The nuance (why the gate looks like it should already allow this): the !== undefined guard is deliberate — LlmModelInfo.inputModalities is documented as "absent means unknown, while an explicit omission is negative capability" (packages/llm/llm/src/types.ts:294). For pi-ai route models that contract is preserved: the per-model schema exposes input without a default (packages/llm/llm-pi-ai/src/config.ts:295-299, "no answer here → the catalog entry still applies") and resolveModelInfo maps it straight through (adapter.ts:279 inputModalities: [...model.input]). The llm-deepseek catalog schema is the one place the .default(['text']) makes the "unknown" branch unreachable for user-added rows: every editor-created model resolves to explicit text-only.

Consequence for the fix: removing the schema default would not be the right move — it would flip every unconfigured model to "unknown", moving image failures downstream to the wire/API instead of failing the send with a clear code. Your toggle shape is exactly consistent with the fail-closed contract:

• checked → writes ["text", "image"] → gate admits
• unchecked → removes the key → schema default ['text'] → explicit text-only, still rejected

That keeps the default posture (a model that never declared image capability keeps rejecting) while giving users the single explicit way to opt a gateway model in. One suggestion for the copy: the gate already emits reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' (commands.ts:323), so the client can branch the toast on that reason to point at the model editor instead of a generic "switch models".

Scope note: since the editor is the "direct DeepSeek adapter's advisory model catalog" (its header comment), the same UI gap does not cover pi-ai custom providers — there the equivalent declaration is input: ['text','image'] in the route's modelOverrides/models entry, which the same gate honors (adapter.ts:279). If the toggle later generalizes beyond the DeepSeek adapter, it maps 1:1 to pi-ai's input field.

I did not see your fork diff, but the 7-file shape (editor + zh/en copy + tests) sounds right-sized. Happy to look at the diff if you drop a link here.

[Dxboy266]

Update / correction after reproducing the reporter's actual setup:

The report covers TWO adapters, and only the first half was in the original post:

1. llm-deepseek — as described above; fixed by the image-input checkbox (commit 77a7582).
2. llm-pi-ai — the reporter's actual route. A hand-declared route falls back to DEFAULT_INPUT = ["text"] (packages/llm/llm-pi-ai/src/config.ts) unless it names defaultInput, and the route editor never exposed that field either. The built-in catalog even knows some of these models are image-capable (e.g. gpt-5.6-sol ships input: ["text","image"] in the pi-ai data), but a custom route key does not read the built-in entry, so the default wins.

commit 5b0378c adds a Text (fixed) + Images checkbox pair to a declared pi-ai route's editor, writing defaultInput: ["text","image"] or ["text"].

Diff (both commits): Dxboy266/deepseek-harness@master...feat/model-image-input-toggle

[argszero]

Thanks for the correction — it completes the picture, and I verified the pi-ai half against 4e84901e64. There is one documented design decision behind this that is worth surfacing, plus a route-shape nuance that sharpens where your checkbox is (and isn't) the remedy.

The route defaultInput default is deliberate. The implemented design note .agents/notes/implemented/architecture/2026-08-12-pi-ai-route-default-input-modalities.md documents exactly this chain — entry input → installed catalog entry → route defaultInput, which "itself defaults to [text]" — and the reasoning is fail-closed: no OpenAI-compatible listing endpoint reports modalities, so the only safe floor is text, and "over-claiming admits and persists an image before the provider can reject it". The same note also records the original decision that no configuration surface edits input (the remedy was the settings document, one line: defaultInput: [text, image]). Your checkbox is a legitimate product evolution of that decision — it writes the route-level field the resolver already reads — but it does change a documented stance, so flagging it as a design question (not just a bug fix) helps the maintainers.

Verified resolution chain (packages/llm/llm-pi-ai/src/catalog.ts:838):

input: declaredInput(entry.input) ?? base?.input ?? [...request.defaultInput]

• base = the installed catalog entry, keyed by provider id (catalogModels(provider), catalog.ts:186). So the catalog fallback exists — but only for routes whose key is a built-in pi-ai provider. A custom route key (catalogModels(customKey) = empty map) has base undefined, and the route defaultInput ['text'] wins even when the model id exists in some built-in provider's catalog. That confirms your "a custom route key does not read the built-in entry" — precisely because the lookup is per-provider, not per-model. This is the reporter's case, and it means your defaultInput checkbox is the correct remedy for it (nothing else in the chain can answer).
• For a catalog route (key = built-in provider), base.input outranks the route default — e.g. gpt-5.6-sol on its home route already resolves image-capable from the catalog without any UI. Your checkbox only appears when ownsIdentity, so it doesn't add a footgun there; and because the route value is a fallback, a checked route can never strip a catalog vision model's input (the asymmetry the note specifically protects against).

Fix shape checks out against the schema. Writing ['text','image'] / ['text'] matches profile.defaultInput (config.ts:324) and the route-level empty-list refusal (config.ts:444-445 refuses [] because "nothing below it can answer instead") — your always-present Text checkbox with the disabled state is exactly right. For tests, packages/llm/llm-pi-ai/tests/catalog.spec.ts already covers "a route default answering an undeclared model while an entry still outranks it" — your added cases for the custom-route-key shape (no catalog base → defaultInput wins; checkbox writes the explicit pair) slot in beside it.

One connection worth making: CDeZT's #5366 (native dynamic model-catalog for CLIProxyAPI) is the dynamic remedy for the same pain — a fetched rich catalog (with input_modalities/supported_reasoning_levels) would make custom route keys read real capabilities instead of the ['text'] floor. Your checkbox is the static-document remedy; the two are complementary, and the reporter's case is evidence for both.