Optional AI visual descriptions via a pluggable inference fleet (local + cloud) — offer to implement

[jeroentrappers]

Optional AI visual descriptions via a pluggable inference fleet (local + cloud) — and an offer to build it

Following up on #26690 and #12900. In #26690 the two concerns raised were that a VLM is
"too heavy for the majority of users" and that descriptions would "duplicate CLIP's
semantic search." Both are fair — so this proposal is designed specifically around them,
and I'm offering to implement it and submit staged PRs matching whatever architectural
decisions you prefer.

The idea (opt-in, off by default)

A Visual Description capability: a background job that, per asset, calls a configured
inference provider and stores a human-readable description (and optionally tags) on the
asset. Crucially, "inference" is just an endpoint — not a bundled model.

On "too heavy" → it ships no default compute

• Off by default. Zero impact unless an admin enables it. No new mandatory model, no new
  default job, no RAM/VRAM footprint.
• Provider-pluggable, over one wire format (OpenAI-compatible /v1/chat/completions
  with image input), so local and cloud are the same code path:
  • No hardware → a cloud endpoint (OpenRouter/OpenAI/Gemini), pennies per 1k images.
  • Privacy-first → a small local model (Qwen3-VL-2B/4B, SmolVLM, Florence-2) via
    Ollama/llama.cpp — modest hardware, some CPU-only.
  • Power user → a fleet across several machines.
• So the users who don't want it pay nothing; the users who do pick their own cost model.

On "duplicates CLIP" → different job entirely

CLIP is opaque retrieval. A description is readable text you can display, edit,
translate, export to EXIF, feed to a screen reader, and full-text search for exact
entities. It doesn't compete with smart-search — it consumes CLIP/OCR/faces as context
and emits something new:

Use case	CLIP can't	Description can
Accessibility / alt-text (a11y)	vectors aren't readable	screen readers announce real text
Slideshow / album captions (#8710, #7058)	no text to show	a sentence under the photo
Exact-entity / compositional search	fuzzy on names, counts, negation	"invoice from T.L.P.C", "Anna and Ben on a beam"
Portability / export	vectors don't travel	text in description → EXIF, survives export
Editing / downstream (tags, memories, translation)	—	user-editable source text

How it fits Immich

Reuses the existing pieces — nothing exotic:

• A new VisualDescription BullMQ job, chained after SmartSearch/FaceRecognition/OCR
  so it can inject named people + capture date + reverse-geocoded location + existing OCR
  into the prompt → "Anna and Ben in Ghent, 2023" instead of "two children". This is the
  concrete answer to "duplicates CLIP": it uses your existing signals.
• Stored in asset.description (already searchable + EXIF-exported) with a provenance
  flag so AI text never overwrites manual/EXIF text and is reversible/re-runnable.
• A server-side provider layer (TS) — so cloud + remote-local endpoints need no change
  to immich-machine-learning. (A first-party bundled small ONNX VLM could be a later,
  separate option — out of scope for v1 to keep the "no default weight" promise.)
• Video: sample a few frames, tile into one montage image, describe the progression — a
  temporal caption at one image's inference cost.

The "fleet" part

A provider router with per-provider concurrency caps, health checks, a circuit breaker
that auto-reloads a stuck/crashed local model and re-admits a recovered host, least-loaded
dispatch, and per-provider budgets for cloud spend. A dead/asleep backend can't stall the
queue.

Why I can move fast on this

I've built and run this exact pipeline in production against a ~120k-photo library
(non-English output): a self-healing multi-host fleet (local Macs + NVIDIA + cloud-ready via
the OpenAI-compatible path), with people/date context injection, structured JSON output,
video montages, and idempotent/resumable backfill. So the reference design is validated —
including the failure modes (stalls, OOM on tight GPUs, laptop sleep) and their mitigations.
A full architecture write-up is available; happy to share.

Ask

Are you open to this under these constraints (opt-in, endpoint-only, no default compute,
complementary to CLIP)? If so, a few decisions would let me start on Phase 1:

1. Store in asset.description + provenance flag (my preference), or a dedicated field?
2. Provider layer server-side (my preference — enables cloud with no ML-service change), or
   extend immich-machine-learning?
3. Detected objects → the existing tags system, or inline in the description only?
4. v1 endpoint-only (no bundled model), or do you want a first-party small local option too?

I'll shape the work into small, reviewable PRs:
Phase 1 — opt-in single provider, image description, backfill, provenance;
Phase 2 — fleet router (multi-provider + health/circuit-breaker/budgets) + context injection;
Phase 3 — tags + video.

Happy to adjust any of this to fit how you'd like it built.

[github-actions[bot]]

This discussion has automatically been closed as it is likely a duplicate. We get a lot of duplicate threads each day, which is why we ask you in the template to confirm that you searched for duplicates before opening one. If you're sure this is not a duplicate, please leave a comment and we will reopen the thread if necessary.

[jeroentrappers]

Thanks for the auto-triage — I did search first, and this intentionally builds on #26690 and #12900 rather than restating them.

Those threads are open-ended discussions about whether to add VLM descriptions. This is a concrete, opt-in architecture aimed squarely at the two objections raised in #26690:

• "Too heavy for the majority of users" — it's off by default and endpoint-only. Inference is any OpenAI-compatible provider — local (Ollama / llama.cpp / vLLM) or cloud (OpenRouter / OpenAI / Gemini) — so Immich ships no default model and no new compute. No-GPU users point it at a cheap cloud endpoint or simply don't enable it.
• "Duplicates CLIP's semantic search" — descriptions are a different modality: readable, editable, EXIF-exportable, screen-reader-friendly text that consumes CLIP/OCR/faces as context rather than competing with retrieval (a11y, slideshow captions, exact-entity search, export).

It also includes an offer to implement it and submit small, reviewable PRs, backed by a validated reference design (self-healing multi-host inference fleet, people/date/location context injection, video-frame montages).

Could this be reopened? If you'd prefer to consolidate, I'm equally happy to move the proposal and the implementation offer into #26690 — just say which you'd like.