Skip to content

Custom/aliased OpenAI-compatible providers: attachment caps may degrade to text-only #2741

@simonferquel-clanker

Description

@simonferquel-clanker

Problem

Providers like xAI, Mistral, Nebius, Ollama, and other custom OpenAI-compatible endpoints route through pkg/model/provider/oaistream with whatever Provider string is set in the model config (e.g. "xai", "mistral", "ollama"). The modelcaps system looks these up in models.dev, but the provider IDs in models.dev may not match the provider strings used in docker-agent configs.

For example:

  • An xAI model config with provider: xai, model: grok-2-vision-1212 produces a lookup for "xai/grok-2-vision-1212"
  • An Ollama config with provider: ollama, model: llava produces "ollama/llava"

If models.dev doesn't have those exact keys, capability detection silently falls back to text-only and image/PDF attachments are dropped.

Additionally, user-configured aliases (via display_model) are already handled correctly by ModelCapsID() (which uses Model, not DisplayOrModel()), but the underlying provider identifier mapping to models.dev still depends on the config's provider field matching models.dev exactly.

Possible fixes

  • Maintain a mapping table from docker-agent provider strings to models.dev provider IDs
  • Allow per-model capability overrides in the agent config YAML
  • Warn (rather than silently drop) when a model ID is not found in models.dev, so users can diagnose the issue

Context

Identified during review of PR #2738 (fix: pass fully-qualified provider/model ID to modelcaps.Load).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)priority:mediumNormal priority, standard sprint workstatus/needs-designRequires architectural discussion or design review

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions