fix(#563): back off image-refresh on a flapping-readiness deployment

[divyasinghds]

Closes #563.

Fix

Record per-target attempt count before the restart and back off after maxRefreshAttempts (default 3) consecutive failures, so a flapping-readiness deployment is no longer re-restarted every cycle.

Files

• client/templates/image-refresh-cronjob.yaml, client/values.yaml, client/values.schema.json

Validation

helm template, sh -n, schema JSON valid.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes production auto-refresh behavior for jobs-manager rollouts and deployment annotations; mis-tuned thresholds or annotation read failures could delay image updates until manual intervention.

Overview
Stops the image-refresh CronJob from endlessly rollout restarting jobs-manager when digest drift is detected but rollouts never settle (readiness flaps). After maxRefreshAttempts consecutive failed attempts (default 3, configurable via values/schema), it skips further restarts, sets tracebloc.io/refresh-flap-detected, and expects an operator to clear tracebloc.io/refresh-attempt or fix the image; a successful rollout clears both counters.

Hardens the refresh script (#626): get_annotation fails closed on kubectl/jq errors (no silent “0 attempts”), adds --request-timeout=15s on deployment reads/annotates/restarts, and resets attempt/flap annotations in a separate annotate before recording digest annotations so a hung digest write cannot leave a false permanent flap lockout.

Chart version 1.9.24 → 1.9.25.

^{Reviewed by Cursor Bugbot for commit be73029. Bugbot is set up for automated code reviews on this repo. Configure here.}

[divyasinghds]

Simplified the image-refresh flap handling per maintainer direction. Dropped the auto-resume / cooldown / signature-reset machinery (BACKOFF_RESUME_AFTER, cooldown/settled-tick counting, resume-to-MAX-1, and the annotate_args signature target-change reset) in favor of a plain consecutive-failure counter:

• increments (persisted before the restart) when a rollout is issued but does not settle;
• resets to 0 only on a genuinely successful settled rollout;
• after MAX_REFRESH_ATTEMPTS in a row it stops restarting, logs a WARN, and annotates tracebloc.io/refresh-flap-detected for a human / monitoring — no auto-resume.

Kept the #563 core: settled-guard, digest-change detection, fail-closed annotation reads, and --request-timeout=15s on every non-watch kubectl call. Net -30 lines. Validated with helm lint, helm template, sh -n/dash -n on the embedded script, and a tick-by-tick stubbed-kubectl simulation (flap→stop+flag, healthy→reset, read-error→skip, new-digest→refresh, unsettled→short-circuit).

🤖 Claude Code

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bcc4803. Configure here.}

[shujaatTracebloc]

Reviewed the code and all checks: LGTM.

• Consecutive-failure counter (refresh-attempt) persisted BEFORE the restart so a rollout that exits the tick under set -e still advances it; resets to 0 only on a genuinely settled rollout. Sound flap guard (#563).
• Success reset is its own bounded annotate BEFORE the digest-record write (CID 3729110045), so a hung/failed digest annotate can no longer strand an elevated count and permanently trip the flap lockout.
• get_annotation reads FAIL-CLOSED: kubectl captured separately from the jq pipe (no pipefail here) and returns non-zero on either error, so a read failure skips the tick rather than collapsing to "0 attempts" and re-restarting a flapping deployment.
• All kubectl calls now --request-timeout-bounded; schema + values.yaml expose maxRefreshAttempts (default 3); Chart bumped 1.9.22→1.9.23.

Green across the board; no unresolved Bugbot/review threads. Satisfies #563.