llm-pi-ai: custom provider entries silently resolve to text-only when the model is actually multimodal (no auto-merge of provider catalog modalities, undocumented per-model input field, no vision badge in the model selector)

[derekkage]

The problem

For custom provider entries under llm-pi-ai.providers.*.models[] (OpenRouter, NVIDIA NIM, Empero, custom openai-completions routes), a model's resolved inputModalities resolve through a chain:

1. the per-model input field on the entry,
2. the catalog base?.input,
3. the provider's defaultInput,
4. ["text"] (the default).

The image-admission gate (dsh-api-session-controller, error session/attachment-invalid, reason MODEL_DOES_NOT_SUPPORT_IMAGES) trusts only that resolved value, so an undeclared multimodal model is silently treated as text-only and attachments are refused at send time with "select a vision-enabled model." The model itself sees fine; the harness metadata is what's wrong.

In one settings.yaml audit I hit five routes in this state, all confirmed multimodal against the vendor's own docs:

• z-ai/glm-5.3-flash (OpenRouter + Empero) — Z.ai's VLM guide and "Z.ai's first natively multimodal open-weights model".
• qwen/qwen3.7-flash (OpenRouter) — OpenRouter model card, "Alibaba's cheap vision model lands on OpenRouter".
• minimaxai/minimax-m3 (NVIDIA) and minimax/minimax-m3:free (OpenRouter) — MiniMax M3: 1M context, native multimodality.
• google/gemma-4-26b-a4b-it:free and google/gemma-4-31b-it:free (OpenRouter) — Gemma 4 is multimodal (vision input documented in the model card).
• nvidia/nemotron-3-nano-omni-30b-a3b-reasoning (NVIDIA) — Nemotron Omni is multimodal (omni = text + image + audio; the -omni suffix in the model id is the signal).

Each needed a hand edit of ~/.dsh/settings.yaml to add input: [text, image]. None of this is surfaced in the GUI; users hit the admission error and have to reverse-engineer the metadata schema. The modlens plugin even special-cases glm-5.3-flash as a native vision model (its VISION_ID regex lists it explicitly), which is independent evidence that the model is multimodal and the harness is just uninformed.

Proposed change

1. Auto-merge the provider's live catalog modalities into the resolved model. OpenRouter's /api/v1/models returns architecture.modality / input_modalities per model; NVIDIA NIM's catalog returns modality per model. When llm-pi-ai builds the catalog base for a model, merge input_modalities into the resolved inputModalities unless the user has explicitly overridden via the per-model input field. The fallback chain becomes: per-model input > catalog modality > defaultInput > ["text"]. This makes "no declaration = catalog default" instead of "no declaration = text-only," and turns the most common case (the model is in the catalog, the catalog is correct) into zero-config.

2. Document the per-model input field. The field exists in the zod schema for the llm-pi-ai package and accepts a MODALITIES union (text, image, audio, video) — but it isn't in the dsh readme, isn't in a settings example, and isn't surfaced in the GUI. Document it next to (or instead of) the llm-deepseek plugin's inputModalities field. The spelling difference between the two provider plugins (input vs inputModalities) is a real footgun and not obvious from the file structure.

3. Surface vision-capable routes in the model selector. A small badge/column showing whether each entry declares image input, and whether the live catalog confirms it. This is the discoverability fix that turns "select a vision-enabled model" from a guessing game into something users can see and act on. It also surfaces the catalog-vs-declaration drift so power users know which entries to override.

Scope check

Pure metadata plumbing; no change to provider adapters, the image admission gate, or the image read/transform pipeline. The admission gate already handles inputModalities: [text, image] correctly — it's the gate that refused these five cases, and it would simply stop refusing once the metadata is right. Every existing override mechanism (per-model input, defaultInput) stays intact; the change is purely additive to the fallback chain.

Alternatives considered

• Keep hand-declaring per-model input in settings.yaml. Works, not discoverable, and breaks every new multimodal model a user adds until they notice the admission error.
• A third-party "fixer" plugin that injects modalities. Sidesteps the harness; doesn't help users without the plugin; duplicates the metadata layer.
• A separate provider plugin (like llm-deepseek) per custom provider. Doesn't scale; llm-pi-ai is the right home and already has the catalog fetch surface to extend.

Environment

• dsh 0.1.2-alpha.5 (web profile, ~/.dsh/profiles/web)
• @deepseek-ai/dsh-llm-pi-ai (current installed version)
• Windows 11
• llm-pi-ai.providers.{nvidia, openrouter-live, empero} mounted in settings.yaml; models span text-only and multimodal in each provider.

---

My notes for you (not part of the issue)

• The MODALITIES union and the per-model input field are in the installed llm-pi-ai package's zod schema (the zod field is input: z.array(z.union(MODALITIES)); the default defaultInput is ["text"]). They are functional, just not surfaced.
• The llm-deepseek plugin uses inputModalities (plural) instead of input. That's a different plugin with a different field name for the same concept, which is exactly the cross-plugin footgun the issue calls out.
• The catalog fetch already exists for llm-pi-ai providers (the base layer is the catalog fetch). The change is to read architecture.modality / equivalent from the fetched model and merge it into the resolved inputModalities — small, contained, no new dependency.
• The vision badge in the model selector is the discoverability fix that would have prevented all five of our cases. Even without (1), (3) would make (1) easy to drive.
• Evidence for each model is cited inline; if the dsh maintainer wants more, the vendor docs and model cards are the primary sources.
• I did NOT include Mistral Small 4 119B (2603) or the StepFun Step-3.5/3.7 flash models in the issue body, because the evidence for those is lineage-based rather than direct ("Mistral Small 3.1+ added vision" / "StepFun has vision variants") and I didn't want to add a claim the maintainer would have to chase. If you can confirm those from vendor docs, we can add them in a comment.

[argszero]

Source-verified against master (d347e70390) — your chain analysis is exact, and this report is the third independent confirmation of the same declaration-surface gap. The two prior ones:

• Model settings UI never exposes the catalog schema existing inputModalities — image-capable custom models are silently rejected #5367 (Dxboy266, 2026-09-02): the llm-deepseek twin — its catalog schema .default(['text']) (index.ts:172) erases the "unknown" branch entirely, and the Models editor only writes id/name/contextWindow/maxTokens, so an image-capable model is silently fixed to text-only. Fork had an editor-checkbox implementation.
• The c249 follow-up (same thread) established the pi-ai side you are now on: defaultInput defaults to ['text'] (llm-pi-ai/src/config.ts:324, DEFAULT_INPUT :76), and the resolution chain is exactly what you derived — per-model input (no schema default, :298) → catalog entry base?.input → route defaultInput → ['text'] (catalog.ts:889: input: declaredInput(entry.input) ?? base?.input ?? [...request.defaultInput]). An archived architecture note (2026-08-12, "pi-ai route default input modalities") documented the fail-closed rationale: no listing endpoint reported modalities, so over-claiming would "persist the image first, then get rejected by the provider".

Your key new fact changes that calculus. OpenRouter /api/v1/models and NVIDIA NIM catalogs DO return per-model modality — so "no listing endpoint" is no longer true for the two biggest custom-route providers. That makes your proposal 1 (merge catalog input_modalities into the resolved model, chain becomes per-model input > catalog modality > defaultInput > ['text']) architecturally sound where it matters most. Two nuances from the existing code to fold in:

1. Catalog lookup is keyed by provider id (catalog.ts, catalogModels(provider) per provider) — which is exactly why your five routes fell through to defaultInput: glm-5.3-flash etc. are absent from the built-in catalog for that route's provider, so base is undefined even when the id exists in another provider's catalog. "Merge the live catalog modalities" must therefore consult the vendor's live catalog for the route's provider id — which is precisely the data OpenRouter/NIM now serve per model. The merge point would be where base is resolved for custom routes (catalog.ts:889's base?.input slot), not the built-in catalog.
2. Keep the user-override precedence. Your chain already does this (input field first) — and the schema comment at config.ts:295 ("no explicit default, unlike the route's defaultInput") confirms the per-model input was designed as the override seam. The one-line settings.yaml hand-edit you did per route (input: [text, image]) is the documented-remedy path; auto-merge just removes the need for it on catalog-known models.

Your proposals 2 and 3 are exactly right and cheap: the per-model input field exists in the zod schema but is invisible (no readme, no settings example, no GUI surface) — and the input vs inputModalities spelling split between the two provider plugins is a genuine footgun that has now confused reporters in all three threads. The GUI badge (declared vs catalog-confirmed) is the discoverability fix that turns "select a vision-enabled model" from a guessing game into a visible state.

One framing note for whoever picks this up: the three threads are one family with two distinct fixes — (a) declaration-surface: make editors/documentation expose the modality fields (#5367's fork shape + your proposals 2–3), and (b) resolution-source: let catalog-confirmed modalities fill the gap when undeclared (your proposal 1, now enabled by vendor catalogs). (b) is the bigger behavioral change and deserves its own design pass because it flips the fail-closed default for catalog-known models; (a) is uncontroversial. The modlens VISION_ID special-casing you cite is good independent evidence — regex-maintained vision lists are exactly the kind of thing that should be replaced by the catalog data, not extended.