Skip to content

image-refresh settled-guard: flapping-readiness deployment can be re-restarted each tick #563

Description

@shujaatTracebloc

Found by the post-incident installer/chart audit (runtime cronjobs). Narrow residual of the #546 settled-guard fix (the core fix is correct).

Problem

client/templates/image-refresh-cronjob.yaml:130-133 (guard) + :179-205: the recorded annotation advances only after a rollout status that fully succeeds within the 10m timeout.

Failure scenario

A deployment whose readiness oscillates (becomes Ready, then crashes after warmup, right around the timeout boundary) can (a) fail rollout status on the acting tick → no annotation, then (b) be observed "settled" on a later tick while recorded != latestrollout restart re-issued (~every 15 min). Requires genuinely flapping readiness; narrow.

Proposed fix

Back off after N consecutive failed refresh attempts for the same target digest (record attempt+digest, not just success), so a flapping deployment isn't re-restarted every tick.

Severity: Tier 3 — low. Confidence: PLAUSIBLE.

Metadata

Metadata

Labels

devopsDevOps / CI-CD / build & deploywork-type:bugDefect or regression

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions