🤖 Generated by the Agentic Engineer
Evidence
Prod GitOps delivery has been frozen since 2026-08-07T00:49Z. The root flux-system
OCIRepository reports:
Ready=False reason=VerificationError
SourceVerified=False reason=VerificationError
failed to verify the signature using provider 'cosign keyless':
no matching attestations: unsupported: multiple identities are not supported at this time
The live resource carries a three-entry spec.verify.matchOIDCIdentity. cosign's keyless
verification rejects a multi-entry list outright and fails closed for the whole set, so the
source refuses every artifact.
Consequence: every merge_group deploy times out waiting for the pushed artifact to be served,
fails, and the PR is evicted from the merge queue. Nothing has merged to main since
2026-08-07T00:36Z. Observed on #3004 (evicted twice) and #3011 (evicted twice).
The cluster itself is healthy and still serving the last good revision — this is a delivery
freeze, not an outage.
The actual problem
main is already correct. Both halves carry the single-entry alternation form:
The corrected config cannot reach the cluster, because the trust configuration for the root
delivery source is itself delivered through that source:
ksail.prod.yaml / flux-instance.yaml → Kustomization infrastructure-controllers
→ rendered by flux-operator
→ onto OCIRepository flux-system
→ which must verify the artifact carrying that very fix
infrastructure-controllers is stuck Progressing, so the fix is pinned in Git. Field-manager
inspection confirms spec.verify is owned by flux-operator (Apply) and not by the routine
deploy path, so a redeploy cannot rewrite it either. It is a closed loop: the delivery channel
cannot carry its own trust config.
Clearing the current freeze needs a one-shot out-of-band write of the corrected single-entry
verify block onto the live root OCIRepository. That is a manual maintainer action and is being
handled separately; this issue is about making the loop structurally impossible.
Proposed fix
The deploy already solves exactly this class of problem for the pull credential: a
🔑 Reassert Flux and consumer GHCR pull credential step writes it out-of-band, before the
reconcile that depends on it. The root source's verify block has the same
bootstrap-circularity and no such step.
Reassert spec.verify on the root OCIRepository from ksail.prod.yaml out-of-band in the deploy,
ahead of the reconcile step, using the same pattern and the same field manager as the operator so
there is no second writer.
Acceptance criteria
Rough size: S–M. Related: #3007 (CI does not check that a cosign matcher actually verifies the
artifact it guards) — this incident is exactly the failure #3007 predicted.
Evidence
Prod GitOps delivery has been frozen since 2026-08-07T00:49Z. The root
flux-systemOCIRepository reports:
The live resource carries a three-entry
spec.verify.matchOIDCIdentity. cosign's keylessverification rejects a multi-entry list outright and fails closed for the whole set, so the
source refuses every artifact.
Consequence: every
merge_groupdeploy times out waiting for the pushed artifact to be served,fails, and the PR is evicted from the merge queue. Nothing has merged to
mainsince2026-08-07T00:36Z. Observed on #3004 (evicted twice) and #3011 (evicted twice).
The cluster itself is healthy and still serving the last good revision — this is a delivery
freeze, not an outage.
The actual problem
mainis already correct. Both halves carry the single-entry alternation form:k8s/providers/hetzner/infrastructure/controllers/flux-instance/flux-instance.yaml(thekustomize patch, since fix(security): enforce cosign verify on the live root OCIRepository #3001)
ksail.prod.yaml(spec.workload.flux.verify, the bootstrap path)The corrected config cannot reach the cluster, because the trust configuration for the root
delivery source is itself delivered through that source:
infrastructure-controllersis stuckProgressing, so the fix is pinned in Git. Field-managerinspection confirms
spec.verifyis owned byflux-operator(Apply) and not by the routinedeploy path, so a redeploy cannot rewrite it either. It is a closed loop: the delivery channel
cannot carry its own trust config.
Clearing the current freeze needs a one-shot out-of-band write of the corrected single-entry
verify block onto the live root OCIRepository. That is a manual maintainer action and is being
handled separately; this issue is about making the loop structurally impossible.
Proposed fix
The deploy already solves exactly this class of problem for the pull credential: a
🔑 Reassert Flux and consumer GHCR pull credentialstep writes it out-of-band, before thereconcile that depends on it. The root source's verify block has the same
bootstrap-circularity and no such step.
Reassert
spec.verifyon the root OCIRepository fromksail.prod.yamlout-of-band in the deploy,ahead of the reconcile step, using the same pattern and the same field manager as the operator so
there is no second writer.
Acceptance criteria
spec.verifybefore the reconcile step.manual intervention (the decisive test — it must fail before the change and pass after).
contention with
flux-operator.Rough size: S–M. Related: #3007 (CI does not check that a cosign matcher actually verifies the
artifact it guards) — this incident is exactly the failure #3007 predicted.