fix(distributed): orchestrator resilience — auto-upgrade routing, worker bind-wait, RAG-init crash, log spam

[localai-bot]

Summary

Four small but impactful fixes for distributed-mode orchestration. Triggered by a real cluster (5-node K3s + a NATS-attached Jetson Orin / Spark) where the orchestrator pod was crash-looping on every chat that touched RAG.

• fix(distributed): route auto-upgrade through BackendManager (a39c0b6)
  UpgradeChecker.runCheck already routed the check through the active BackendManager, but the auto-upgrade branch right below it called gallery.UpgradeBackend directly using the frontend's SystemState. In distributed mode the frontend has no backends installed locally, so every reported upgrade failed with backend "<name>": backend not found. The fix routes the upgrade (and the post-upgrade re-check) through bm.UpgradeBackend(...) so it fans out to workers via NATS. Local mode still falls back to gallery.UpgradeBackend.

• fix(worker): wait for backend gRPC bind before replying to backend.install (97a8c74)
  The supervisor used to wait 4s (20 × 200ms) for the backend's gRPC to answer a HealthCheck and then log a warning and reply Success anyway. On Jetson Orin first-boot CUDA init that wasn't enough — the frontend's first LoadModel dial got connect: connection refused instead of a real error. The wait window is now 30s, and on deadline-exceeded the supervisor stops the half-started process, recycles the port, and returns an error with the backend's stderr tail.

• fix(distributed): bump LocalAGI/LocalRecall — RAG-init no longer crashes the host (a39c0b6, paired with the dep bumps)
  LocalRecall's NewPersistent{Chrome,LocalAI,Postgres}Collection used to call os.Exit(1) on init failure (acceptable for the standalone CLI, catastrophic for a long-running embedder). When a chat lazily initialized RAG and the embedding worker was unreachable, os.Exit(1) killed the orchestrator pod. LocalRecall now returns errors (mudler/LocalRecall@6138c1f), LocalAGI surfaces them as a nil collection (mudler/LocalAGI@e83bf51), and the existing RAGProviderFromState returns (nil, nil, false) — the same path agents take when no RAG is configured.

• fix(nodes/health): skip stale-marking already-offline nodes (5c10fb1)
  The health monitor re-emitted Node heartbeat stale + Marking stale node offline + MarkOffline every cycle for nodes that were already offline, flooding the logs whenever an operator intentionally took a node down. Skip the staleness branch when the node is already StatusOffline / StatusUnhealthy.

Test plan

• go build ./core/application/... ./core/services/agentpool/... ./core/services/nodes/... ./core/cli/... ./core/http/endpoints/openai/...
• go vet clean on the same set
• go test ./core/services/nodes/ (all node tests pass, ~94s)
• go test ./core/cli/
• Roll the orchestrator pod onto an image built from this branch + bumped LocalAGI dep and confirm:
  • chat that triggers RAG init survives an unreachable embedding worker (no pod restart, falls back to "no RAG available")
  • first cold-load on the Jetson Orin returns a real error (or succeeds) instead of connection refused
  • dgx-spark stays offline without log spam
  • auto-upgrade actually upgrades workers instead of failing with "backend not found"