fix(distributed): persist per-model load info so reconciler survives frontend restart

[localai-bot]

Summary

After a LocalAI frontend container restart (rolling upgrade, env-var change, etc.), the Replica Reconciler logs failed to scale up replica ... no load info every 30s for each affected model and cannot maintain min_replicas until someone sends a fresh inference request to that model. Production impact: in a window where a frontend has restarted but no traffic has hit a given model, a dying worker will not be replaced.

Root cause

router.scheduleAndLoad already persisted the (backendType, pb.ModelOptions) pair, but it stamped it onto the NodeModel row of the loading replica. That row gets deleted by every healthy-row-removal path: MarkOffline reaping stale workers, RemoveAllNodeModelReplicas on stop, RemoveNodeModel on probe failure or stale health check, evictLRUAndFreeNode. When the last replica row goes away, the only copy of load info goes with it. GetModelLoadInfo's scan over NodeModel then returns gorm.ErrRecordNotFound, and the reconciler has nothing to replicate.

Fix

New dedicated ModelLoadInfo table keyed by model_name, decoupled from NodeModel. Three small hooks:

1. NodeRegistry.UpsertModelLoadInfo(ctx, modelName, backendType, optsBlob) writes the per-model row with ON CONFLICT (model_name) DO UPDATE (last-write-wins under concurrent multi-frontend dispatch, matching the existing per-replica blob semantics).
2. router.scheduleAndLoad calls both the existing per-replica SetNodeModelLoadInfo (backward compat / hot-path) and the new per-model UpsertModelLoadInfo after a successful load.
3. NodeRegistry.GetModelLoadInfo now reads the per-model row first and falls back to the legacy NodeModel-blob scan so a rolling upgrade does not regress already-loaded models that were stamped only the old way.

Migration: the new model_load_infos table joins the existing AutoMigrate call under the same advisory lock; no separate migration step.

Test plan

• go build ./core/services/nodes/... ./pkg/model/... ./core/services/distributed/... succeeds
• go test -race -count=1 ./core/services/nodes/... passes (233 specs, 7 new)
• Registry-level integration specs cover: survival across RemoveAllNodeModelReplicas, ON CONFLICT last-write-wins, legacy NodeModel-blob fallback, empty-source ErrRecordNotFound, empty-model-name rejection
• Reconciler-level integration spec proves recovery: with min_replicas=2, NodeModel rows wiped, ModelLoadInfo row present, one reconcile tick calls the scheduler twice (before the fix: zero calls, "no load info" warning)
• Manual e2e against a multi-frontend deployment - reproduce the original symptom, apply the patch, confirm the reconciler scales replicas up on its own after a frontend restart (no fresh inference request required)

Caveats

• Concurrent dispatch by two frontends of the same model with different opts converges last-write-wins on model_load_infos.model_opts_blob. That is identical to the existing per-NodeModel-row semantics; if stronger ordering is ever needed, the row carries updated_at.
• The blob is the marshalled pb.ModelOptions proto from the dispatch path. If a proto field is renamed/removed in a backward-incompatible way, stale rows would replay obsolete options; standard proto compatibility rules apply.
• Schema migration is gorm AutoMigrate under the existing advisory lock; no explicit roll-back is provided. Dropping the table on downgrade requires manual SQL (consistent with how the other distributed tables are managed today).

Follow-up to #9976 (distributed-mode observability + middleware refactor).

Assisted-by: Claude:claude-opus-4-7[1m]