amd64 and arm64 1.28.2 kubernetes images fail the cosign signature verification

[weseven]

What happened?

When following the image verification guidelines (as in https://kubernetes.io/docs/tasks/administer-cluster/verify-signed-artifacts/), most 1.28.2 amd64 and arm64 images fail the verification with cosign.

E.g.:

[root@local ~]# cosign verify registry.k8s.io/kube-apiserver-amd64:v1.28.2 \
  --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com \
  --certificate-oidc-issuer https://accounts.google.com \
  | jq .
Error: no matching signatures
main.go:69: error during command execution: no matching signatures

This is the cosign version:

[root@local ~]# cosign version
  ______   ______        _______. __    _______ .__   __.
 /      | /  __  \      /       ||  |  /  _____||  \ |  |
|  ,----'|  |  |  |    |   (----`|  | |  |  __  |   \|  |
|  |     |  |  |  |     \   \    |  | |  | |_ | |  . `  |
|  `----.|  `--'  | .----)   |   |  | |  |__| | |  |\   |
 \______| \______/  |_______/    |__|  \______| |__| \__|
cosign: A tool for Container Signing, Verification and Storage in an OCI registry.

GitVersion:    v2.2.0
GitCommit:     546f1c5b91ef58d6b034a402d0211d980184a0e5
GitTreeState:  clean
BuildDate:     2023-08-31T18:52:52Z
GoVersion:     go1.21.0
Compiler:      gc
Platform:      linux/amd64

I've used this script to iterate and verify:

curl -Ls "https://sbom.k8s.io/$(curl -Ls https://dl.k8s.io/release/stable.txt)/release" \
  | grep "SPDXID: SPDXRef-Package-registry.k8s.io" \
  | grep -v sha256 | cut -d- -f3- | sed 's/-/\//' | sed 's/-v1/:v1/' \
  | sort > images.txt
input=images.txt
while IFS= read -r image
do
  if cosign verify "$image" --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com --certificate-oidc-issuer https://accounts.google.com  2>&1 | grep error &> /dev/null;
  then
    echo "$image - FAILED signature verification"
  else
    echo "$image - PASSED signature verification"
  fi

This is the output:

registry.k8s.io/conformance-amd64:v1.28.2 - FAILED signature verification
registry.k8s.io/conformance-arm64:v1.28.2 - PASSED signature verification
registry.k8s.io/conformance-ppc64le:v1.28.2 - PASSED signature verification
registry.k8s.io/conformance-s390x:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-apiserver-amd64:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-apiserver-arm64:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-apiserver-ppc64le:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-apiserver-s390x:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-controller-manager-amd64:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-controller-manager-arm64:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-controller-manager-ppc64le:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-controller-manager-s390x:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-proxy-amd64:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-proxy-arm64:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-proxy-ppc64le:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-proxy-s390x:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-scheduler-amd64:v1.28.2 - PASSED signature verification
registry.k8s.io/kube-scheduler-arm64:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-scheduler-ppc64le:v1.28.2 - FAILED signature verification
registry.k8s.io/kube-scheduler-s390x:v1.28.2 - PASSED signature verification
registry.k8s.io/kubectl-amd64:v1.28.2 - FAILED signature verification
registry.k8s.io/kubectl-arm64:v1.28.2 - FAILED signature verification
registry.k8s.io/kubectl-ppc64le:v1.28.2 - PASSED signature verification
registry.k8s.io/kubectl-s390x:v1.28.2 - PASSED signature verification

What did you expect to happen?

I would expect all official images passing the verification with the latest cosign version.

How can we reproduce it (as minimally and precisely as possible)?

cosign verify registry.k8s.io/kube-apiserver-amd64:v1.28.2
--certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com
--certificate-oidc-issuer https://accounts.google.com

Anything else we need to know?

No response

Kubernetes version

v1.28.2, other image verification fails also for 1.28.1 images for

Cloud provider

OS version

No response

Install tools

No response

Container runtime (CRI) and version (if applicable)

No response

Related plugins (CNI, CSI, ...) and versions (if applicable)

No response

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/sig release

[saschagrunert]

cc @kubernetes/release-managers

[xmudrii]

This is as expected because the signing process has been broken for a few months. See kubernetes/release#2962 for more details

[saschagrunert]

Closing in favor of kubernetes/release#2962