fix(controller): stop hot-spinning on unreachable file:// model sources (closes #405)

[Defilan]

Summary

Closes #405. When a Model's spec.source is a file:// URL pointing at a path that exists on the metal-agent host but not inside the controller pod (the canonical Mac kind / k3s + out-of-cluster agent topology), the reconciler entered a tight loop: every iteration returned an error, controller-runtime's rate-limited workqueue requeued at ~5ms initial backoff ramping into hundreds of reconciles per second, and a Mac kind cluster pinned a CPU core at ~295% for 35 hours before anyone noticed.

Root cause: returning a non-nil error to controller-runtime invokes the rate limiter, which then dominates the schedule and eats the RequeueAfter interval that was supposed to space retries.

Fix

New isUnrecoverableFetchError helper in source.go detects fs.ErrNotExist and fs.ErrPermission (the canonical "this won't heal without operator intervention" errors). When the fetch fails with one of these, the reconciler:

• still marks the Model Failed with a clear Degraded condition
• still emits the failure metric
• but returns ctrl.Result{RequeueAfter: 5*time.Minute}, nil (note the nil) so controller-runtime treats this as a "successful, recheck later" reconcile rather than a transient error

Steady-state cost: one reconcile every 5 minutes until the operator fixes the spec. Recoverable errors (network timeouts, transient apiserver issues) keep the original err+RequeueAfter shape so the rate limiter still provides backoff for genuinely transient problems.

Testing

Unit (6 new cases on isUnrecoverableFetchError):

• nil error → false
• fs.ErrNotExist directly → true
• fs.ErrPermission directly → true
• The exact fmt.Errorf("failed to open local model file: %w", os.Open-ENOENT) wrap shape from copyLocalModel → true (the Model controller hot-spins on file:// sources unreachable from controller pod #405 reproduction; load-bearing assertion)
• Generic non-fs error → false
• Wrapped non-fs error → false
• Double-wrapped fs.ErrNotExist → true (errors.Is walks the wrap chain)

Reconcile-level (2 new + 2 updated):

• New: "no rate-limited tight retry" test asserts err == nil, RequeueAfter == 5m, Requeue == false, and Failed + Degraded/CopyFailed condition for a missing file:// source. The nil err assertion is the load-bearing regression guard.
• New: "repeated reconciles stay on fixed interval" loops three times against the same broken source and asserts each iteration returns the same shape (this catches future changes that take a different code path on subsequent reconciles).
• Updated: should set Failed with CopyFailed for nonexistent local file flipped from asserting err.HaveOccurred() to asserting err == nil (the new behavior).
• Updated: should not leave partial file at final path on fetch failure same flip; the atomic-rename guarantee is now checked against status conditions instead of err return.

Documentation

Added a "file:// caveat for hybrid topologies" paragraph to the Model.spec.source GoDoc explaining the controller-pod-readability constraint, what happens when the path is unreachable (Failed + 5min backoff, not hot-spin), and the two workarounds (pre-stage on pvc:// or use the equivalent https://huggingface.co/.../<file>.gguf URL). make manifests regenerated the Model CRD description.

Compatibility

No CRD field changes. Behavior change is limited to the error-return shape on unreachable sources: pre-fix, controller-runtime saw err != nil and tight-retried; post-fix, controller-runtime sees err == nil and respects the 5m RequeueAfter. Operators won't notice unless they were debugging the hot-spin (in which case CPU drops from 200-300% to flat).

Closes #405

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!