feat(api): add GET /v1/models/capabilities endpoint#10687
Merged
Conversation
Additive superset of /v1/models that enriches each model entry with the capabilities it supports plus its input/output modalities (text / image / audio / video). Clients that only understand /v1/models are unaffected -- they simply never call the new route. Audio and video *input* are derived from the model's multimodal limits (vLLM limit_mm_per_prompt), which no single usecase FLAG expresses. That gap is exactly why a plain capability list is insufficient and this enriched endpoint exists: an attachment router can now decide whether an image/audio/video file can go to the active model directly, or must be converted/transcribed first. Capability derivation lives in core/config as the single source of truth (ModelConfig.Capabilities / InputModalities / OutputModalities / VisionSupported / ...); the Ollama capability surface now delegates to it instead of keeping a parallel copy. Vision is gated on chat/completion capability so a MediaMarker hydrated onto a non-chat model (e.g. a pure ASR/TTS backend) no longer reports a false vision capability. Read-only listing: no new FLAG_* flag, reuses the existing `models` swagger tag, and intentionally exposes no MCP admin tool (there is nothing to manage conversationally). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
GET /v1/models/capabilities(+/models/capabilities), a purely additive superset of/v1/models. It returns the same set of models but enriches each entry with the capabilities the model supports plus its input/output modalities:{ "object": "list", "data": [ { "id": "qwen2.5-omni", "object": "model", "capabilities": ["chat", "vision", "tools"], "input_modalities": ["text", "image", "audio"], "output_modalities": ["text"] }, { "id": "parakeet", "object": "model", "capabilities": ["transcript"], "input_modalities": ["audio"], "output_modalities": ["text"] } ] }Clients that only understand
/v1/modelsare unaffected — they simply never call the new route.Why
A client (e.g. a desktop/CLI assistant) needs to know, before sending a request, whether the active model can accept an image / audio / video attachment — and route it directly, or convert/transcribe first. Today that information isn't exposed on any single endpoint:
/v1/modelsreturns IDs only./api/showsurface reportsvisionbut not audio/video/transcript.config-metadata/autocomplete/models:<usecase>filters one usecase at a time.Critically, audio and video input cannot be expressed by a usecase flag at all — they're derived from the model's multimodal limits (vLLM
limit_mm_per_prompt), andFLAG_VIDEOmeans video generation (diffusers), not video input. That gap is the reason a plain capability list is insufficient and this endpoint carries explicitinput_modalities/output_modalities.How
core/config/model_capabilities.go:ModelConfig.Capabilities(),InputModalities(),OutputModalities(), plusVisionSupported()/AudioInputSupported()/VideoInputSupported(). The Ollama capability surface now delegates to these instead of keeping a parallel copy.VisionSupported()(notHasUsecases(FLAG_VISION), which the guess heuristic paints onto any chat model) and is gated on chat/completion capability — so aMediaMarkerhydrated by config defaults onto a non-chat model (e.g. a pure ASR/TTS backend) no longer reports a falsevisioncapability. (A unit test caught exactly this.)listVisibleModelNameshelper extracted fromListModelsEndpoint, so filtering (filter,excludeConfigured) and the per-user model allowlist behave identically to/v1/models..well-known/localai.jsonanddocs/content/features/api-discovery.md+whats-new.mdupdated; swagger regenerated.Checklist notes
FLAG_*flag → nocapabilities.jschange needed.modelsswagger tag → no new/api/instructionsarea.list_installed_modelsalready offers capability filtering for the MCP surface).Testing
core/config,core/http/endpoints/openai, andcore/http/endpoints/ollamasuites pass;gofmtandgo vetclean. New unit tests cover the capability/modality derivation and the endpoint (auth-disabled path; the per-user allowlist branch is shared verbatim with/v1/models).make test-coverage-check) could not complete in my environment:core/http/app_test.gohard-binds127.0.0.1:9090, which was occupied by an unrelated local service. This is environmental and independent of the diff — please let CI run the full gate.Assisted-by: Claude:claude-opus-4-8 [Claude Code]