fix(distributed): reject wrong-model requests at the backend

[localai-bot]

Problem

In distributed mode the controller caches a NodeModel row naming a backend's host:port. A worker can recycle a stopped backend's gRPC port for a different model's backend. probeHealth verifies liveness, not identity, so the probe succeeds against whatever now occupies the port and the request is dispatched to the wrong backend. The caller gets a silent wrong-model answer.

Nothing in the request path could catch it:

• PredictOptions had no model field, so identity crossed the wire only in ModelOptions.Model at LoadModel — and the cached-hit path in Route issues no LoadModel.
• Every backend's "model not loaded" guard checks a nil handle (backend/python/vllm/backend.py:392-395, backend/cpp/llama-cpp/grpc-server.cpp:2183-2188, backend/go/parakeet-cpp/goparakeetcpp.go:295-297). A process holding a different model passes all of them, so IsModelNotLoaded never matched, reconcile never fired, and the stale row was never dropped.
• Backend processes are single-model, so "wrong backend" means "a live process holding a different model" — and it answers.

Fixes #10952. PR #10968 (per-key port affinity) closes the common paths but cannot close this class: while liveness is the only check, a stale row over a re-bound port is indistinguishable from a valid one.

Approach

Add PredictOptions.ModelIdentity and enforce at the point of use, rather than in probeHealth — which short-circuits on a 30s probeCache keyed nodeID|addr and would skip verification during exactly the window that matters.

• Controller: populated in gRPCPredictOpts from ModelConfig.Model.
• Backends: pkg/grpc/server.go (27/27 Go backends), an interceptor in backend/python/common (36/36 Python backends, no per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers.
• Router: reconcile drops the stale replica row on a mismatch, so the next request reloads correctly.

Four RPCs take PredictOptions and all four are covered: Predict, PredictStream, Embedding, TokenizeString.

Why this cannot false-reject

ModelOptions(c, ...) passes model.WithModel(c.Model), which becomes ModelOptions.Model at LoadModel. gRPCPredictOpts(c, ...) receives the same config value in the same function a few lines later — verified at all four call sites (llm.go:136/177, embeddings.go:46/58, tokenize.go:32/39, face_embed.go:23/33). Equal by construction, not by convention.

stageModelFiles rewrites ModelFile, MMProj, LoraAdapter, DraftModel, CLIPModel, Tokenizer and AudioPath — but never Model. That matters given the managed-artifact work, which moves ModelFile to a snapshot directory.

Compatibility

Empty means skip, enforced on both sides:

• New controller + old backend: field unknown, proto3 ignores it. Unprotected, as expected.
• Old controller + new backend: field arrives empty, backend skips. Not optional — tests/e2e-backends/backend_test.go drives real backends with bare pb.PredictOptions{Prompt: ...} at 8+ sites.
• A backend loaded by an old controller has no recorded identity, so it skips too.
• The C++ server synthesizes PredictOptions internally on the ASR path — empty, so it skips.

A unit spec pins "empty identity is accepted", so a future tightening fails there before it breaks the e2e suite.

Deliberately not in scope

TTSRequest.model / SoundGenerationRequest.model are NOT validated, despite already carrying a model string. FileStagingClient.TTS/.TTSStream (file_staging_client.go:219,260) already rewrite in.Model into a worker-local absolute path, while the load-time value is untranslated — so in distributed mode those two strings already differ today. Comparing them would reject valid requests, in the exact configuration this fix targets, and only for TTS.

This also explains backend/go/piper/piper.go:29-31, which passes opts.Model straight to synthesis and stores nothing at load: it needs the translated path.

Extending identity to those RPCs requires a separate field carrying the untranslated value. Recorded in the proto comment so it is not rediscovered the hard way.

Known coverage gap (not a regression)

Rows are keyed by ModelID() but identity compares c.Model, so two configs sharing one Model with different ModelID() are a false accept. Sending ModelID() instead would false-reject universally, since it is not what LoadModel receives. c.Model is the only value with the structural guarantee above.

Why IsModelMismatch needs code AND sentinel

Unlike the neighbouring helpers, which match on code or message, this one requires NOT_FOUND and a sentinel substring.

backend/python/insightface/backend.py:127 returns NOT_FOUND "no face detected" from Embedding, which is a PredictOptions RPC. A code-only check would make reconcile drop a healthy replica row on every faceless image — turning this fix into a new bug. FailedPrecondition was unusable (claimed by IsModelNotLoaded), as was InvalidArgument (llama.cpp returns it ~15 times for ordinary bad parameters).

Documented in code, with a spec asserting IsModelMismatch is false for the insightface error, so a future "simplification" fails a test.

Scope correction during implementation

The long tail was originally scoped out as "not on the hot path". Checking it properly: privacy-filter implements no PredictOptions RPC at all, and kokoros answers all four with unimplemented. The tail was one backend — ds4, which has real Predict/PredictStream/TokenizeString — so it is included rather than left as the only exposed backend without the guard.

Testing

Every guard was verified red-first against the mistake it prevents, not merely written and observed green:

• IsModelMismatch implemented code-only first, so the insightface spec failed on behavior: 3 Failed. Adding the sentinel turned it green.
• The Python interceptor wired below get_auth_interceptors' early return first — the natural mistake, since it returns [] when LOCALAI_GRPC_AUTH_TOKEN is unset, which is the default. Red output: identity enforcement must be installed even with gRPC auth off (the default); got []. That got [] is the silent no-op across all 36 Python backends. Moving it above the token check: Ran 15 tests — OK.
• Bare-PredictOptions acceptance passed from the start (correct shape — tightening it is what would break e2e); the red case was the mismatch being served when it should reject.
• End-to-end: load identity A, predict identity B, assert mismatch and assert reconcile drops the row, for both unary and streamed calls.

ok  pkg/grpc              (race) 2.141s
ok  pkg/grpc/grpcerrors   (race) 1.022s
ok  core/backend                 0.544s
ok  core/services/nodes          175.373s
Python: Ran 15 tests — OK

make lint: 0 issues. Coverage 53.4% → 53.6%, baseline untouched.

C++ helpers could not be compiled in this environment; each was spliced out of the real source and compiled standalone against stubs, exercising all four rules (skip on empty request, skip on empty loaded, reject on mismatch, serve on match). Residual risk there is the call sites, not the logic.

Follow-ups

• Transparent same-request retry on mismatch (deferred: it means re-running routing mid-request).
• Model identity for the other modalities (GenerateImage, TranscriptRequest, RerankRequest, DetectOptions, VADRequest) — they share the exposure but carry no model field, so each needs its own.
• Unrelated one-liner: root .gitignore:44 has a bare mock-backend pattern that matches the directory, so git will not descend into it even for tracked files (tests/e2e/mock-backend/main.go needed git add -f). Should be /mock-backend or mock-backend/mock-backend.

---

🤖 Generated with Claude Code