Skip to content

mirror: copy cosign .sig OCI artifacts alongside release images#4779

Merged
yjst2012 merged 1 commit intomasterfrom
b-spaithanka/mirror-sig-artifacts
Apr 17, 2026
Merged

mirror: copy cosign .sig OCI artifacts alongside release images#4779
yjst2012 merged 1 commit intomasterfrom
b-spaithanka/mirror-sig-artifacts

Conversation

@shubhadapaithankar
Copy link
Copy Markdown
Collaborator

@shubhadapaithankar shubhadapaithankar commented Apr 16, 2026

Summary

  • Mirror pipeline (pkg/mirror/mirror.go) now copies cosign .sig OCI artifacts alongside each release and component image
  • For digest-based references (component images), the digest is extracted from the reference string; for tag-based references (release images), the manifest is fetched to compute the digest
  • .sig copy failures are logged at debug level and treated as non-fatal, so older unsigned releases are unaffected

Context

Incident: ITN-2026-00112
IcM: 780900379
Jira: ARO-26171

OCP 4.21 ships a ClusterImagePolicy that enforces cosign signature verification for all pulls of quay.io/openshift-release-dev/ocp-release. In disconnected ARO clusters that depend on �rosvc.azurecr.io, the mirror pipeline never copied .sig OCI artifacts, causing SignatureValidationFailed during z-stream upgrades within 4.21.x.

Test plan

  • Unit tests pass for
    epoFromReference, sigReference, digestFromReference
  • Run mirror pipeline in INT to validate .sig artifacts are copied to arointsvc
  • Verify via skopeo inspect that .sig tags exist in destination ACR for mirrored releases
  • Confirm impacted customer cluster (IcM 780900379) can complete 4.21.7 upgrade after production mirror run

The mirror pipeline copies release images and component images from quay.io to arosvc.azurecr.io but never copies the cosign .sig OCI artifacts (tagged as sha256-<digest>.sig). OCP 4.21 enforces cosign signature verification via ClusterImagePolicy for ocp-release pulls, so disconnected clusters that depend solely on arosvc fail with SignatureValidationFailed during z-stream upgrades.

After each successful image copy, the worker now attempts to also copy the corresponding .sig artifact from the source registry. For digest-based references (component images), the digest is extracted from the reference string. For tag-based references (release images), the manifest is fetched to compute the digest. If a .sig artifact does not exist (older unsigned releases), the failure is logged at debug level and does not affect the overall mirror result.

Fixes: ARO-26171
@shubhadapaithankar
Copy link
Copy Markdown
Collaborator Author

/azp run ci

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@yjst2012
Copy link
Copy Markdown
Collaborator

yjst2012 commented Apr 17, 2026

okay, it looks all references flowing through the mirror pipeline are digest-based, so please ignore my previous comment below:
I'm not sure I get the full context
in the PR description:
"for tag-based references (release images), the manifest is fetched to compute the digest"

~~However digestFromReference() is purely string-based — it looks for @sha256: in the reference. The release image is enqueued as:~~
~~```~~
~~ch <- &work{~~
~~    tag:          "release",~~
~~    srcreference: srcrelease,  // e.g. "quay.io/.../ocp-release:4.21.7-x86_64"~~
~~    ...~~
~~}~~
~~```~~
~~Since srcrelease is a tag-based reference (no @sha256:...), digestFromReference will return an error, copySigArtifact will bail out, and the .sig for the release image itself is silently skipped. Only component images (which use digest-based refs ~~from ~~the image stream) will get their .sig mirrored.~~
~~Is this expected?~~

@yjst2012 yjst2012 merged commit 37d7841 into master Apr 17, 2026
31 checks passed
@yjst2012 yjst2012 deleted the b-spaithanka/mirror-sig-artifacts branch April 17, 2026 04:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants