Split out of #552. The requests-proxy hardcoded-Always bug is fixed in its own PR; this ticket tracks the harder, fleet-affecting part: making the always-running control-plane pods survive an offline Docker/WSL restart without breaking how they update. It needs a deliberate design decision, not a one-line pull-policy flip.
The constraint
jobs-manager + pods-monitor are updated by the image-refresh CronJob, which polls the registry and does kubectl rollout restart — this relies on imagePullPolicy: Always to pull the new digest. Flipping them to IfNotPresent silently breaks auto-updates.resource-monitor is not touched by image-refresh; its Always is pure offline-fragility (updates only via a chart release / ref change).- Offline-safe (
IfNotPresent/digest+cache) and auto-update-via-rollout-restart (Always) are mutually exclusive unless the reference itself changes on update (i.e. a digest).
Options
A — prodDigest pin (mirror resolve-ingestor-digest.sh). Add prodDigest/prodPin for jobsManager/podsMonitor/resourceMonitor (and align requestsProxy), render repo@digest + IfNotPresent where the pin applies, and retire image-refresh's poll-restart for the pinned images (updates come via a release-time digest re-resolve + Chart bump that auto-upgrade propagates). Restart-safe + reproducible in prod; dev/staging stay floating+Always (still offline-fragile). Needs registry digest resolution + a release step. This is what #552 originally proposed.
B — digest-on-update. Switch the images to IfNotPresent and change image-refresh to pin the new digest via kubectl set image repo@digest (instead of rollout restart). Offline-safe and auto-updating in dev+prod, self-contained in the chart. Must handle the helm upgrade --reset-then-reuse-values interaction: a chart release re-renders :tag and reverts the set-image, so image-refresh re-pins on its next tick (a brief, self-healing window). Out-of-band set image is also invisible to Helm (drift).
Recommendation
Decide A vs B before implementing. A matches the existing ingestor.prodDigest pattern and is lower-risk but prod-only; B fixes dev too (which is where the incident client runs) but reworks the refresh contract. Also decide whether requests-proxy (which runs the jobs-manager image) should track the same pin as jobsManager.
Refs: #552, #565 (installer-side), the requests-proxy fix PR.
Severity: Tier 1 — breaks-on-restart. Confidence: CONFIRMED.
Split out of #552. The
requests-proxyhardcoded-Alwaysbug is fixed in its own PR; this ticket tracks the harder, fleet-affecting part: making the always-running control-plane pods survive an offline Docker/WSL restart without breaking how they update. It needs a deliberate design decision, not a one-line pull-policy flip.The constraint
jobs-manager+pods-monitorare updated by the image-refresh CronJob, which polls the registry and doeskubectl rollout restart— this relies onimagePullPolicy: Alwaysto pull the new digest. Flipping them toIfNotPresentsilently breaks auto-updates.resource-monitoris not touched by image-refresh; itsAlwaysis pure offline-fragility (updates only via a chart release / ref change).IfNotPresent/digest+cache) and auto-update-via-rollout-restart (Always) are mutually exclusive unless the reference itself changes on update (i.e. a digest).Options
A — prodDigest pin (mirror
resolve-ingestor-digest.sh). AddprodDigest/prodPinforjobsManager/podsMonitor/resourceMonitor(and alignrequestsProxy), renderrepo@digest+IfNotPresentwhere the pin applies, and retire image-refresh's poll-restart for the pinned images (updates come via a release-time digest re-resolve + Chart bump that auto-upgrade propagates). Restart-safe + reproducible in prod; dev/staging stay floating+Always(still offline-fragile). Needs registry digest resolution + a release step. This is what #552 originally proposed.B — digest-on-update. Switch the images to
IfNotPresentand change image-refresh to pin the new digest viakubectl set image repo@digest(instead ofrollout restart). Offline-safe and auto-updating in dev+prod, self-contained in the chart. Must handle thehelm upgrade --reset-then-reuse-valuesinteraction: a chart release re-renders:tagand reverts the set-image, so image-refresh re-pins on its next tick (a brief, self-healing window). Out-of-bandset imageis also invisible to Helm (drift).Recommendation
Decide A vs B before implementing. A matches the existing
ingestor.prodDigestpattern and is lower-risk but prod-only; B fixes dev too (which is where the incident client runs) but reworks the refresh contract. Also decide whetherrequests-proxy(which runs the jobs-manager image) should track the same pin asjobsManager.Refs: #552, #565 (installer-side), the requests-proxy fix PR.
Severity: Tier 1 — breaks-on-restart. Confidence: CONFIRMED.