fix(providers): add per-model attachment capability override (#2741)#3205
Merged
Conversation
Custom and aliased OpenAI-compatible providers (xai, mistral, nebius, ollama, and user-defined proxies) resolve attachment capabilities through a models.dev lookup keyed on the config provider/model string. When that pair is absent from models.dev (uncatalogued provider, local Ollama tags, or a dropped model version), detection silently fell back to text-only and dropped image and PDF attachments (issue #2741). Add a typed `capabilities: {image, pdf}` override on the model config. When set it is authoritative and bypasses the models.dev lookup; when omitted, behaviour is unchanged. Also emit a deduplicated diagnostic when a model is absent from models.dev so the degraded behaviour is no longer silent. The override is resolved once per client (base.Config.CapsOverride) and threaded through the five attachment-conversion paths: oaistream (openai chat, dmr), openai responses, anthropic, bedrock, gemini. modelinfo.ResolveCaps takes plain booleans so modelinfo keeps no dependency on the config package.
Reconcile the per-model attachment capability override (#2741) with main's attachment caps-injection refactor. Conflicts and resolution: - pkg/model/provider/oaistream/messages.go, attachments.go: keep main's caps-injection internals (convertMessagesWithCaps / convertMultiContentWithCaps / convertDocumentWithCaps and the public ConvertMessagesWithCaps); re-add the override only at the public ConvertMessages / ConvertMultiContent entry points, resolving via modelinfo.ResolveCaps instead of LoadCaps. The now-dead convertDocument store wrapper is dropped. - pkg/model/provider/dmr/client.go: default attachment capabilities to those declared in provider_opts (main's attachmentCaps); a config capabilities override, when present, takes precedence.
4b2a969 to
8f24cef
Compare
dgageot
approved these changes
Jun 26, 2026
aheritier
added a commit
that referenced
this pull request
Jun 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Custom and aliased OpenAI-compatible providers (xai, mistral, nebius, ollama, and user-defined proxies) resolve attachment capabilities through a models.dev lookup keyed on the config
provider/modelstring (base.Config.ID()). When that pair is absent from models.dev,modelinfo.LoadCapsswallows the "not found" error and returns conservative text-only capabilities, soattachment.Decidedrops every image and PDF part with only a generic warning. Affected models lose attachments silently.Fixes #2741.
Verified against the live models.dev catalogue:
ollama-cloud, no local tags)ollama/llavadroppedgrok-2-vision-1212is gone (drifted to grok-4.x)gpt-4odroppedWhy not a provider mapping table
The issue listed a provider to models.dev mapping table as one option. Live
api.jsonshows it would fix almost nothing:xai,mistral,nebius,minimax,github-copilotalready match models.dev exactly (identity mapping),ollamahas no catalogue entry at all, and neither model-absence (grok-2-vision-1212) nor user-defined providers can be fixed by a provider rename. The vertexai precedent (withModelsDevProvider) works there only because those models do exist in models.dev under a different key, which does not hold here.What this does
Implements two of the issue's three options: the per-model escape hatch and the non-silent diagnostic.
capabilities: {image, pdf}. When set it is authoritative and skips the models.dev lookup; when omitted, behaviour is unchanged.pkg/config/latestCapabilitiesConfigplus optionalCapabilitiesfield onModelConfig;isShorthandOnlyupdatedpkg/modelinfoCapsOverrideplusResolveCaps(store, id, override); once-per-model miss diagnostic inLoadCapspkg/model/provider/baseConfig.CapsOverride()builds the override from configagent-schema.json,examples/capability-overrides.yamlmodelinfo.ResolveCapstakes plain booleans, somodelinfokeeps no dependency on the config package (layering preserved). The override is resolved once per client and passed down; a nil override reproduces prior behaviour at every call site.Example:
Issue expectations mapping
capabilities)Backward compatibility
Capabilitiesis optional and nil by default; existing configs are unaffected. Onlypkg/config/latestis touched (frozen config versions untouched).Tests
pkg/model/provider/oaistreamrepropkg/modelinfoResolveCaps plus diagnosticpkg/config/latestpkg/model/providerend-to-endRegistry.Newto the built clientCapsOverride()pkg/model/provider/baseCapsOverride()mappingNotes and residual design points
needs-design; the chosen direction is the override plus the diagnostic. The field name and the "override fully wins" semantics are open for maintainer review.