Floating-tag drift detection: verify clusters pick up SHA changes under a floating tag

[sfreudenthaler]

Parent epic: #35693 · split out of #36520 (concrete-scope item 2)

Background

The evergreen-tracks promotion engine (cicd/evergreen-tracks/) re-points floating Docker tags (latest, standard, trailing) to a new release digest. Re-pointing a tag does not by itself guarantee a running cluster actually rolls to the new SHA — a deployment pinned to a floating tag only picks up the change on its next pull/roll.

Scope

Build a tool — modeled on the one in the OVH cloud-control cluster — that:

• Detects when the SHA behind a floating tag changes (tag moved to a new digest).
• Verifies the target cluster(s) actually pick up the new digest (pods roll / running image digest matches the tag's current digest).
• Surfaces a clear signal when a tag has moved but a cluster has not converged.

Out of scope

• The promotion engine / tag-move mechanics themselves (Floating Docker tags for Release Tracks: latest/standard/trailing promotion engine #36160, Rebuild track promotion engine to honor 2-week maintenance-window cadence #36520).
• Control-plane / API work (Workstream B).

Acceptance criteria

• Given a floating tag whose digest changed, the tool reports whether each target cluster is running the new digest.
• A tag-moved-but-cluster-not-converged state is detectable and alertable.
• Validated against the dotcms/dotcms-test sandbox.

Context

Split out of #36520, which reduced the promotion cadence to manual, human-approved dispatch. Drift detection is orthogonal to how/when tags move, so it lives on its own.

[sfreudenthaler]

Implementation spec + plan (lives in infrastructure-as-code)

Spec written to kubernetes/evergreen-reconciler/SPEC.md — draft PR dotCMS/infrastructure-as-code#7998. The issue stays here in core for OKR/roadmap tracking; the spec + all implementation live in the IaC repo where every artifact and reviewer is.

Locked design — windowed, git-committing, in-cluster reconciler:

• The AWS fleet is ArgoCD GitOps (StatefulSets pinned to immutable digests, auto-sync master), not the OVH Deployment/Keel model — so the OVH image-reconciler doesn't port; the reference is the mechanism only.
• One in-cluster CronJob per cluster; native spec.schedule+spec.timeZone is the maintenance window: biweekly Wed 21:00–23:00 in each cluster's local tz (anchored 2026-07-22 via an in-script week-parity gate — customer standard/trailing versions can't move weekly).
• On fire it resolves each subscribed tenant's floating-tag digest (direct to hub.docker.com, bypassing mirror.gcr.io cache lag), rewrites only the image line, and pushes straight to master → ArgoCD rolls. No PR, no human.
• Image-field-scoped: out-of-window config changes (JAVA_OPTS, cpu/mem, ALB) still sync normally at any time — the whole reason for commit-in-window over blanket ArgoCD sync windows.
• Convergence = successful in-window sync; post-push the job asserts pod imageID == committed digest and alerts on non-convergence.
• Subscription: label dotcms.cloud/evergreen-track: latest|standard|trailing; absent = opt-out.

Out of scope (unchanged): promote engine (#36520/#36160), control-plane (Workstream B), and a separate follow-up for the StatefulSet→Deployment / node-clustering question.

Full task list (4 PRs incl. sandbox validation on dotcms/dotcms-test) is in the spec.

[sfreudenthaler]

Done — acceptance criteria met. The evergreen-tracks reconciler (spec'd here; built in dotCMS/infrastructure-as-code starting with dotCMS/infrastructure-as-code#7998 + follow-ons) detects when a floating tag's digest changes, verifies each subscribed env's running pod imageID against the committed digest, and alerts on non-convergence (dotCMS/platform#341). Deployed dry-run to all 4 prod clusters (Sydney/Frankfurt/US/Canada — IaC #8023/#8052/#8053/#8054), convergence poll hardened (#8084). Remaining rollout & hardening (arming PUSH_ENABLED, digest-idempotent runs, App-bot commits) is tracked in sub-epic dotCMS/infrastructure-as-code#8098 — closing this spec/roadmap issue.