Skip to content

feat(api): add GET /v1/models/capabilities endpoint#10687

Merged
mudler merged 1 commit into
masterfrom
feat/models-capabilities-endpoint
Jul 5, 2026
Merged

feat(api): add GET /v1/models/capabilities endpoint#10687
mudler merged 1 commit into
masterfrom
feat/models-capabilities-endpoint

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

What

Adds GET /v1/models/capabilities (+ /models/capabilities), a purely additive superset of /v1/models. It returns the same set of models but enriches each entry with the capabilities the model supports plus its input/output modalities:

{
  "object": "list",
  "data": [
    {
      "id": "qwen2.5-omni",
      "object": "model",
      "capabilities": ["chat", "vision", "tools"],
      "input_modalities":  ["text", "image", "audio"],
      "output_modalities": ["text"]
    },
    {
      "id": "parakeet",
      "object": "model",
      "capabilities": ["transcript"],
      "input_modalities":  ["audio"],
      "output_modalities": ["text"]
    }
  ]
}

Clients that only understand /v1/models are unaffected — they simply never call the new route.

Why

A client (e.g. a desktop/CLI assistant) needs to know, before sending a request, whether the active model can accept an image / audio / video attachment — and route it directly, or convert/transcribe first. Today that information isn't exposed on any single endpoint:

  • /v1/models returns IDs only.
  • The Ollama /api/show surface reports vision but not audio/video/transcript.
  • config-metadata/autocomplete/models:<usecase> filters one usecase at a time.

Critically, audio and video input cannot be expressed by a usecase flag at all — they're derived from the model's multimodal limits (vLLM limit_mm_per_prompt), and FLAG_VIDEO means video generation (diffusers), not video input. That gap is the reason a plain capability list is insufficient and this endpoint carries explicit input_modalities / output_modalities.

How

  • Single source of truth in core/config/model_capabilities.go: ModelConfig.Capabilities(), InputModalities(), OutputModalities(), plus VisionSupported() / AudioInputSupported() / VideoInputSupported(). The Ollama capability surface now delegates to these instead of keeping a parallel copy.
  • Vision is resolved via VisionSupported() (not HasUsecases(FLAG_VISION), which the guess heuristic paints onto any chat model) and is gated on chat/completion capability — so a MediaMarker hydrated by config defaults onto a non-chat model (e.g. a pure ASR/TTS backend) no longer reports a false vision capability. (A unit test caught exactly this.)
  • Handler reuses a new shared listVisibleModelNames helper extracted from ListModelsEndpoint, so filtering (filter, excludeConfigured) and the per-user model allowlist behave identically to /v1/models.
  • .well-known/localai.json and docs/content/features/api-discovery.md + whats-new.md updated; swagger regenerated.

Checklist notes

  • No new FLAG_* flag → no capabilities.js change needed.
  • Reuses the existing models swagger tag → no new /api/instructions area.
  • Read-only listing → no MCP admin tool intentionally (there's nothing to manage conversationally; list_installed_models already offers capability filtering for the MCP surface).

Testing

  • core/config, core/http/endpoints/openai, and core/http/endpoints/ollama suites pass; gofmt and go vet clean. New unit tests cover the capability/modality derivation and the endpoint (auth-disabled path; the per-user allowlist branch is shared verbatim with /v1/models).
  • ⚠️ The full local pre-commit coverage gate (make test-coverage-check) could not complete in my environment: core/http/app_test.go hard-binds 127.0.0.1:9090, which was occupied by an unrelated local service. This is environmental and independent of the diff — please let CI run the full gate.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Additive superset of /v1/models that enriches each model entry with the
capabilities it supports plus its input/output modalities
(text / image / audio / video). Clients that only understand /v1/models
are unaffected -- they simply never call the new route.

Audio and video *input* are derived from the model's multimodal limits
(vLLM limit_mm_per_prompt), which no single usecase FLAG expresses. That
gap is exactly why a plain capability list is insufficient and this
enriched endpoint exists: an attachment router can now decide whether an
image/audio/video file can go to the active model directly, or must be
converted/transcribed first.

Capability derivation lives in core/config as the single source of truth
(ModelConfig.Capabilities / InputModalities / OutputModalities /
VisionSupported / ...); the Ollama capability surface now delegates to
it instead of keeping a parallel copy. Vision is gated on
chat/completion capability so a MediaMarker hydrated onto a non-chat
model (e.g. a pure ASR/TTS backend) no longer reports a false vision
capability.

Read-only listing: no new FLAG_* flag, reuses the existing `models`
swagger tag, and intentionally exposes no MCP admin tool (there is
nothing to manage conversationally).

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler merged commit b0959d4 into master Jul 5, 2026
61 checks passed
@mudler mudler deleted the feat/models-capabilities-endpoint branch July 5, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants