Skip to content

Reassert the root OCI source's verify config out-of-band so a bad value stays recoverable #3014

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

The 2026-08-07 delivery freeze (~35h, cleared by a manual cluster write on 2026-08-08) had a single
structural cause: the trust configuration for the root OCI source is delivered through that
source.
ksail.prod.yaml / flux-instance.yaml → Kustomization infrastructure-controllers
rendered by flux-operator → onto OCIRepository/flux-system — which must verify the artifact
carrying that very config.

Once the live matcher was wrong, main already held the corrected value and could not deliver it.
Both automatic escape routes were void (verified in #3005): cd.yaml shares the deploy-prod
composite and reaches the same reconcile, and ksail cluster update does not own the FluxInstance.
The heal shares the deploy path too, so it failed alongside the deploy (#3005) — there is no second
line of defence. Blast radius reached the repo: every merge_group deploy timed out and evicted its
PR, so nothing could merge either.

The incident is resolved and the loop is not. The same deadlock recurs on any future bad value.

Proposed fix

The deploy already solves this exact class for the pull credential: a
🔑 Reassert Flux and consumer GHCR pull credential step writes it out-of-band, ahead of the
reconcile that depends on it. The root source's spec.verify has the same bootstrap circularity and
no equivalent step.

Add the mirror-image step: reassert spec.verify on the root OCIRepository from the values main
declares, before the reconcile, under the field manager that already owns the resource
(flux-operator) so there is no second writer and no SSA pruning question.

Acceptance criteria

  • The deploy reasserts the root OCIRepository's spec.verify before the reconcile step.
  • Decisive test: with the live verify block set to a multi-entry list, the next deploy repairs
    it with no manual intervention — it must fail before the change and pass after.
  • The reassert writes under flux-operator; no field contention.
  • A green deploy in which the step was skipped is not mistaken for a pass (a skipped step in a
    green deploy is otherwise invisible).

Scope

Recovery only. Prevention — catching a matcher that cannot verify before it ships — is #3007, and
the two are complementary: #3007 stops the bad value being published, this stops a bad value that is
already live from being unrecoverable.

Rough size: S–M. Supersedes the remediation halves of #3005 and #3006, both closed as resolved.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Status
    📥 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions