Skip to content

The root OCI source's cosign trust config is delivered through the source it guards #3013

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

Prod GitOps delivery has been frozen since 2026-08-07T00:49Z. The root flux-system
OCIRepository reports:

Ready=False           reason=VerificationError
SourceVerified=False  reason=VerificationError
  failed to verify the signature using provider 'cosign keyless':
  no matching attestations: unsupported: multiple identities are not supported at this time

The live resource carries a three-entry spec.verify.matchOIDCIdentity. cosign's keyless
verification rejects a multi-entry list outright and fails closed for the whole set, so the
source refuses every artifact.

Consequence: every merge_group deploy times out waiting for the pushed artifact to be served,
fails, and the PR is evicted from the merge queue. Nothing has merged to main since
2026-08-07T00:36Z. Observed on #3004 (evicted twice) and #3011 (evicted twice).

The cluster itself is healthy and still serving the last good revision — this is a delivery
freeze, not an outage.

The actual problem

main is already correct. Both halves carry the single-entry alternation form:

The corrected config cannot reach the cluster, because the trust configuration for the root
delivery source is itself delivered through that source
:

ksail.prod.yaml / flux-instance.yaml  →  Kustomization infrastructure-controllers
                                      →  rendered by flux-operator
                                      →  onto OCIRepository flux-system
                                      →  which must verify the artifact carrying that very fix

infrastructure-controllers is stuck Progressing, so the fix is pinned in Git. Field-manager
inspection confirms spec.verify is owned by flux-operator (Apply) and not by the routine
deploy path, so a redeploy cannot rewrite it either. It is a closed loop: the delivery channel
cannot carry its own trust config.

Clearing the current freeze needs a one-shot out-of-band write of the corrected single-entry
verify block onto the live root OCIRepository. That is a manual maintainer action and is being
handled separately; this issue is about making the loop structurally impossible.

Proposed fix

The deploy already solves exactly this class of problem for the pull credential: a
🔑 Reassert Flux and consumer GHCR pull credential step writes it out-of-band, before the
reconcile that depends on it. The root source's verify block has the same
bootstrap-circularity and no such step.

Reassert spec.verify on the root OCIRepository from ksail.prod.yaml out-of-band in the deploy,
ahead of the reconcile step, using the same pattern and the same field manager as the operator so
there is no second writer.

Acceptance criteria

  • The deploy reasserts the root OCIRepository's spec.verify before the reconcile step.
  • Reverting the live verify block to a multi-entry list is repaired by the next deploy without
    manual intervention (the decisive test — it must fail before the change and pass after).
  • The reassert writes under the field manager that already owns the resource; no field
    contention with flux-operator.
  • A green deploy with the step skipped is not mistaken for a pass.

Rough size: S–M. Related: #3007 (CI does not check that a cosign matcher actually verifies the
artifact it guards) — this incident is exactly the failure #3007 predicted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Status
    ✅ Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions