Verifying signature with Cosign v2 fails due to Transparency Log

[Logankwb]

Bug Description

Version 2 of Cosign enables its "Transparency Log" feature by default. Using Cosign to verify Istio images returns the following error:

# cosign verify --key "https://istio.io/misc/istio-key.pub" istio/pilot:1.17.0
Error: no matching signatures:
signature not found in transparency log
main.go:69: error during command execution: no matching signatures:
signature not found in transparency log

I'm guessing signing the latest Istio images with Cosign v2 would push the signature to the transparency log.

Also the image referred to on the website appears to have moved (I'm happy to raise this on the istio.io repo):

# ./cosign-binary verify --key "https://istio.io/misc/istio-key.pub" gcr.io/istio-release/pilot:1.17.0
Error: image tag not found: GET https://gcr.io/v2/istio-release/pilot/manifests/1.17.0: MANIFEST_UNKNOWN: Failed to fetch "1.17.0" from request "/v2/istio-release/pilot/manifests/1.17.0".
main.go:69: error during command execution: image tag not found: GET https://gcr.io/v2/istio-release/pilot/manifests/1.17.0: MANIFEST_UNKNOWN: Failed to fetch "1.17.0" from request "/v2/istio-release/pilot/manifests/1.17.0".

Version

# cosign version
GitVersion:    v2.0.1
GitCommit:     8faaee4d2b5f65678eb0831a8a3d5990a0271d3a
GitTreeState:  clean
BuildDate:     2023-04-06T19:10:33Z
GoVersion:     go1.20.3
Compiler:      gc
Platform:      linux/amd64

Additional Information

No response

Affected product area

• Ambient
• Docs
• Installation
• Networking
• Performance and Scalability
• Extensions and Telemetry
• Security
• Test and Release
• User Experience
• Developer Infrastructure
• Upgrade
• Multi Cluster
• Virtual Machine
• Control Plane Revisions

[howardjohn]

The missing gcr.io/istio-release/pilot:1.17.0 was a one-of, 1.17.1 and all other versions are present on gcr.io. Not sure about the rest

[howardjohn]

cc @stewartbutler @jacob-delgado

[stewartbutler]

So, it does pass when you add --insecure-ignore-tlog, so it is signed and validated against the key. We just need to figure out why it isn't being uploaded to the tlog.

What's the objective, here? Fix it moving forward, or figure out how to backfill existing signatures to Rekor (which I think is the default transparency log)?

[howardjohn]

It would be nice to backfill if its feasible. If not forward fix.

So I guess - whatever we can :-)

[stewartbutler]

ACK. Let me do some investigation and testing, to see if I can figure out how to do backfill.

[ericvn]

I see the version mentioned is 1.17.0 which I think failed the signing step in some fashion. However, I think 1.17.1 should be OK, although I haven't verified. The above command does fail on all 1.17.x versions.

One other difference might be cosign versions. release-1.17 build-tools has 1.13.1 while the main brach has 2.0.1. I think the only change was adding a -y to skip prompts.

[howardjohn]

1.17.1 also has issues (but less than 1.17.0). As does 1.18.0-alpha.0 which I think uses cosign 2?

[stewartbutler]

One other difference might be cosign versions. release-1.17 build-tools has 1.13.1 while the main brach has 2.0.1. I think the only change was adding a -y to skip prompts.

Ahh, OK, so I wasn't going crazy. I couldn't remember when that swapover happened.

The alpha images use release-builder, right? So they should be getting signed?

Do we have the logs stashed somewhere other than Prow? Or can I rerun another alpha release to get log output?

[ericvn]

For the 1.18 builds (now from the main branch for a few more day(s)), I hopefully fixed with a change in release-builder as we were incorrectly not signing due to a command error: istio/release-builder#1455

[ericvn]

@stewartbutler I only know of the logs in prow, which is where I found the 1.18 issue.

[stewartbutler]

Rgr. Lets get that PR in and rebuild the alpha, check if that fixes it.

I don't think that we should try to backfill, though. Let's just add some documentation saying that prior to 1.18, you have to pass the flag that disables transparency log validation, since it was experimental in the versions of the tool that sign those images.

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2023-04-13. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.