Skip to content

CI does not check that a cosign matcher actually verifies the artifact it guards #3007

Description

@devantler

🤖 Generated by the Agentic Engineer

Evidence

Every check we had passed on a cosign matcher that verified zero artifacts, and it halted all
GitOps delivery on prod for ~5.5 hours (#3005). The config was schema-valid, docs-valid, correctly
targeted, enabled, and constrained a signer — it satisfied a purpose-built validator written for
exactly this control. The defect was only visible in status.conditions on the live cluster, after
the deploy.

#3001 fixes that instance and adds an arity guard so the specific shape cannot return. It also
demonstrates efficacy — the artifact currently on latest is confirmed via the public Sigstore
transparency log to be signed by a publisher the matcher accepts. That demonstration was manual and
one-off.
Nothing asserts it on the next deploy, or the one after.

Affected audience and impact

Anyone operating this cluster. The remaining gap is the general form of the incident: a matcher can be
well-formed and still match nothing — a wrong org, a renamed workflow, a changed trigger ref, a
publisher moved behind a reusable workflow (which relocates the OIDC subject to the called
workflow's path, as the aws and wedding-app OCIRepositories already document). Each of those
produces a control that is present, green, and inert, and the failure lands on the root source where
it is self-locking (#3006).

Expected behaviour

A change to a cosign matcher should fail in CI when the matcher would not verify the artifact it
guards — before it reaches a cluster, not after.

Acceptance criteria

  • CI verifies the configured matcher against the real artifact it guards, and fails the build when
    verification does not succeed. In-repo GITHUB_TOKEN can read the org's packages, so this needs
    no new credential for first-party artifacts.
  • The check reads issuer and subject from the manifest rather than restating them, so the
    thing tested is the thing deployed.
  • A negative control proves non-vacuity: a deliberately wrong subject must fail the check. Without
    this the check is indistinguishable from one that always passes — which is precisely the failure
    class being closed.
  • The transparency-log route is considered as the primary or fallback mechanism. It needed no
    registry credential at all and is what settled fix(security): enforce cosign verify on the live root OCIRepository #3001; cosign verify against GHCR returned
    UNAUTHORIZED from an unprivileged context.
  • Covers the root OCIRepository matcher in both halves (cluster config + FluxInstance patch),
    and states explicitly whether the four app/tenant matchers are in scope.

Size

Medium. The mechanism is understood and the evidence path is proven manually; the work is wiring it
into CI, keeping it non-flaky, and proving it is not vacuous.

Related: #3005 (the incident), #3006 (why it could not self-recover), #3001 (the fix that motivated
this), #2818 (which signer revisions should be acceptable — a related but distinct question).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status
    📥 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions