bug: x509 verification broken

[nsmith5]

Description

The documented x509 certificate verification isn't working as expected. This is broken in two different ways at HEAD (29360f6) and v2.0.0-rc0

HEAD

$cosign verify --certificate-chain bundle.pem --certificate cert.pem  $(cat image)
Error: --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode
main.go:63: error during command execution: --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode

v2.0.0-rc0

$ cosign verify --certificate-chain bundle.pem --certificate cert.pem  $(cat image) --insecure-ignore-sct                                     
Error: no matching signatures:                                                                                                                                               
error verifying bundle: comparing public key PEMs, expected -----BEGIN PUBLIC KEY-----                                                                                       
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEgC9YnGfLG1oNUg7qbVoI9RaCYNmU                                                                                                             
SC8QYw9JCIEdkn+ySEfwRPMVwd8ljiSljFSvw9TWuqxj5WvOMU43wmO9jQ==                                                                                                                 
-----END PUBLIC KEY-----                                                                                                                                                     
, got -----BEGIN CERTIFICATE-----                                                                                                                                            
MIICEzCCAbqgAwIBAgIRAKHhIO8ezAumM0UvolewscEwCgYIKoZIzj0EAwIwJDEi                                                                                                             
MCAGA1UEAxMZTm90RnVsY2lvIEludGVybWVkaWF0ZSBDQTAeFw0yMzAxMTUyMjUz                                                                                                             
NThaFw0yMzAxMTUyMjU5NThaMAwxCjAIBgNVBAMTATEwWTATBgcqhkjOPQIBBggq                                                                                                             
hkjOPQMBBwNCAAQDOAZjN5VZ7wARVO7hoYvf4Ra/UROo/Img1bPIOk5jF1ha+sEp                                                                                                             
duoZ3pKuw7Xv3QCqWWPNNYKr4X5OJoYAbubBo4HkMIHhMA4GA1UdDwEB/wQEAwIH                                                                                                             
gDATBgNVHSUEDDAKBggrBgEFBQcDAzAdBgNVHQ4EFgQUXVBLLUtgD8UljJXEkzXp                                                                                                             
yH8aTywwHwYDVR0jBBgwFoAU5h1Is9hr/ITmJe3qM3aThHsfETAwNgYDVR0RBC8w                                                                                                             
LYEPY29kZUBuZnNtaXRoLmNhhhpodHRwczovL2NvZGUubmZzbWl0aC5jYS8jMTBC
BgwrBgEEAYKkZMYoQAEEMjAwAgECBAVnaXRlYQQkZjQ5YzkwNmYtZGZiZC00NjI2
LWExMWEtMmQyODhlMjYzZmM2MAoGCCqGSM49BAMCA0cAMEQCICPQcmZ4/f+rnERW
a+nTuWcgVSne2X2IzSzAOrJggLh4AiBDKMzfsaDkxY8JAned38JScsA3I0C0tMGk
qivv/JB29w==
-----END CERTIFICATE-----

Version

• v2.0.0-rc0
• HEAD (29360f6)

[haydentherapper]

At HEAD, this is WAI: --certificate-identity or --certificate-identity-regexp is required for verification in keyless mode - You need to specify the identity flags now, but we haven't updated docs for 2.0. That's tracked in another issue.

For the second, it looks like you uploaded a public key to Rekor, but tried to verify the returned bundle with a certificate. This would result in an error, because the verifiers don't match.

[haydentherapper]

To confirm, is this issue saying the docs are broken, or something else? If it's just the docs, can you update #2534?

[nsmith5]

You'd expect that I'd uploaded a public key to Rekor looking at the error, but that isn't the case. The rekor bundle has a certificate (the same as the one I'm verifying with). This can actually be seen from the error message. In

cosign/pkg/cosign/verify.go

Lines 1078 to 1081 in 29360f6

	if !bytes.Equal(pemFirst.Bytes, pemSecond.Bytes) {
	return fmt.Errorf("comparing public key PEMs, expected %s, got %s",
	pemBytes, decodeSecond)
	}

you can see that the value from the rekor bundle is printed second. The first value (the one with the public key) comes from either the the verifier (the certificate passed at the command line) or the signature as you can see in this function:

cosign/pkg/cosign/verify.go

Lines 723 to 737 in 29360f6

	func keyBytes(sig oci.Signature, co *CheckOpts) ([]byte, error) {
	cert, err := sig.Cert()
	if err != nil {
	return nil, err
	}
	// We have a public key.
	if co.SigVerifier != nil {
	pub, err := co.SigVerifier.PublicKey(co.PKOpts...)
	if err != nil {
	return nil, err
	}
	return cryptoutils.MarshalPublicKeyToPEM(pub)
	}
	return cryptoutils.MarshalCertificateToPEM(cert)
	}

.

The keyBytes logic assumes that if co.SigVerifier != nil we must be verifying with a base public key and marshals the public key of the verifier into a PEM, but this isn't the case with we specify to certificate to verify against.

If this is WAI at head I suppose that is fine and the docs just need to be updated, but why should I have to specify details like the issuer etc if I've given you the exact certificate I want you to verify with. It feel like an odd little bit of homework for me to pull that out of the cert and as for it to be checked when I assume this pathway should already check the certificate from the signature and the one passed at the command line match

[haydentherapper]

Ah, this is all scoped to BYO PKI. For the first point, that is reasonable to not require the flags since you’re passing the certificate. The logic would look something like “cert || cert-identity || key” is required. This would also affect verify blob.

However I’m also tempted to say that we should always require the identity flags so that we can have clear documentation around why requiring identity is necessary. Saying it’s not for a subset of cases might be confusing. And there’s edge cases too - if I provide a bundle and not a certificate, I should still require the identity flags. Edit: you’ll also get examples that just extract a certificate from a bundle and pass that, that’s not ideal because users can’t easily see what identity they’re verifying with.

For the second, yea, this seems buggy. Lemme dig through the code later. I think this is because we extract the public key from the certificate rather than have the certificate be the verifier, when —cert is passed. For the keyless case, I’m assuming we make the certificate the verifier, but I’m not sure right now.

[nsmith5]

Yeah exactly, I opened up that feature request issue #2630 because I agree that identity flags + bundle would be a great option, but it feels like a new feature to cosign.

The existing approach of BYO PKI where you specify the exact certificate for verification and we don't have strong opinions on its format (e.g we don't require that custom Fulcio extension for the OIDC issuer to be specified) is something I feel like we should still support. In this case it doesn't seem clear to me that we should require the identity flags because those extensions might not be set by the users PKI right?

[haydentherapper]

Yea, this seems like an issue because we had previously supported certificates like that. A sub command should help this, but for now, I think the easiest option is to say that if you pass a certificate or bundle, we don’t require identity flags. verify-blob wouldn’t require it, only “cosign verify” without a verifier passed to it.

[haydentherapper]

The other option is adding some flag that says my certificate is non-conforming (effectively a BYO flag), but I don’t really think that is great user experience.

cc @znewman01 thoughts on all this?

[haydentherapper]

@nsmith5, do you want to check out #2633 locally and check to see if that mitigates the second issue?

[nsmith5]

@haydentherapper Yup that fixed the second issue!

$ cosign verify --certificate cert.pem --certificate-chain bundle.pem --certificate-identity code@nfsmith.ca --certificate-oidc-issuer https://code.nfsmith.ca/ --insecure-ignore-sct $(cat image)

Verification for code.nfsmith.ca/nsmith/seccon@sha256:86482081aa0e2377bb4d5f97a0099efa9b1ddb714a95d98a11dc2857ceee10f1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"code.nfsmith.ca/nsmith/seccon"},"image":{"docker-manifest-digest":"sha256:86482081aa0e2377bb4d5f97a0099efa9b1ddb714a95d98a11dc2857ceee10f1"},"type":"cosign container image signature"},"optional":{"1.3.6.1.4.1.57264.1.1":"https://code.nfsmith.ca/","Bundle":{"SignedEntryTimestamp":"MEQCIAQb+ORO9oJkjGF2OC4KIeYVDatAudlfZbT1YrSAtIA0AiA5OtW5I9rB2ZV3+HxaHaoCHPophHFUDNWg5m2w+rtcUg==","Payload":{"body":"eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiI4NGVjNDJjMTFkMWI3NjczZmE1MmIzOTk2Mzc3OTQ1YzE0MmQyNmFhZjdjZWFiYzhhZTY3NzA1YThiZGQ1MDBiIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FUUNJQkhDVlJxLy9La2FKcEZublg0cGd3VitMd0VPd3JidTVjQmRteEdrbzl3UUFpQXdpalc5QVM1U0t3TSs4SGxZT2hyZUx5dmlMOFVHZkoxVWFYUnRzT05lM1E9PSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVTlFha05EUVdWUFowRjNTVUpCWjBsUlFXaHpiRVJIUmxoeU9IQjRTMHhaS3pGcFZUUkZha0ZMUW1kbmNXaHJhazlRVVZGRVFXcEJhMDFUU1hjS1NVRlpSRlpSVVVSRmVHeFBZak5TUjJSWGVHcGhWemhuVTFjMU1GcFlTblJhVjFKd1dWaFNiRWxGVGtKTlFqUllSRlJKZWsxRVJYaE9WRWw2VGxSUmVRcE9iRzlZUkZSSmVrMUVSWGhPYWtGM1RVUkJlVTVzYjNkRVJFVkxUVUZuUjBFeFZVVkJlRTFDVFZSQ1drMUNUVWRDZVhGSFUwMDBPVUZuUlVkRFEzRkhDbE5OTkRsQmQwVklRVEJKUVVKSlFYWlhTbmh1ZVhoMFlVUldTVTgyYlRGaFExQlZWMmR0UkZwc1JXZDJSVWROVUZOUmFVSklXa292YzJ0b1NEaEZWSG9LUm1OSVprcFpOR3R3V1hoVmNqaFFWVEZ5Y1hOWksxWnllbXBHVDA0NFNtcDJXVEpxWjJkRlRrMUpTVUpEVkVGUFFtZE9Wa2hST0VKQlpqaEZRa0ZOUXdwQ05FRjNSWGRaUkZaU01HeENRWGQzUTJkWlNVdDNXVUpDVVZWSVFYZE5kMGhSV1VSV1VqQlBRa0paUlVaT2VISnhRVUpPYlRST1JVVXdORXg2ZG1odENrOHdVbXBVTTNOcVRVSTRSMEV4VldSSmQxRlpUVUpoUVVaUFdXUlRURkJaWVM5NVJUVnBXSFEyYWs0eWF6UlNOMGg0UlhkTlJGbEhRVEZWWkVWUlVYWUtUVU15UWtReVRuWmFSMVpCWW0xYWVtSlhiREJoUXpWcVdWbFpZV0ZJVWpCalNFMDJUSGs1YW1JeVVteE1iVFZ0WXpJeGNHUkhaM1ZaTWtWMlNYcEZkd3BLWjFsTFMzZFpRa0pCUjBSMmVrRkNRVkZSV1dGSVVqQmpTRTAyVEhrNWFtSXlVbXhNYlRWdFl6SXhjR1JIWjNWWk1rVjJUVVZKUjBSRGMwZEJVVkZDQ21keFVtdDRhV2hCUVZGUmVVMUVRVU5CVVVsRlFsZGtjR1JIVm1oQ1ExSnRUa1JzYWs5VVFUSmFhVEZyV20xS2EweFVVVEpOYWxsMFdWUkZlRmxUTUhrS1drUkpORTlIVlhsT2FrNXRXWHBaZDBObldVbExiMXBKZW1vd1JVRjNTVVJUVVVGM1VtZEphRUZQUVZwUVVEZHVPV3BrUm5Cd1ZIbEZNelJ5UzBGc05nbzNOa05EWlRGcGVsbHNVa3RUT1VwU1VtVlROMEZwUlVGb1RWbG9jSFpUT0Zkd0wzZDRhbUVyUjI1RlZFeHlVMWNyVW5WYWNTOWFZV3QyV2xCS2JVMVpDakJJUVQwS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvPSJ9fX19","integratedTime":1673826932,"logIndex":11253618,"logID":"c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"}},"Issuer":"https://code.nfsmith.ca/","Subject":"code@nfsmith.ca"}}]

[haydentherapper]

@nsmith5 Been thinking a bit more about how to fix the first issue with requiring the identity flags. The problem is that even if we remove requiring the flag when a certificate or bundle is provided**, what about the case where the signer attaches a BYO PKI certificate without an identity to a container? We would still require the flags, but no identity would be present.

The issue is that certificates can be one of two things within Sigstore: Either an identity-based code signing certificate (issued by Fulcio or a managed PKI that issues certificates that adhere to Sigstore's expectations), or just a vehicle for providing a public key (which some managed PKIs may do because they have existing infrastructure). In the latter case, the identity is irrelevant.

Rather than remove flags, I would propose that we add a flag to disable the requirement, something like allow-all-identities. It's a dangerous flag, but we will make it clear that this is to be used when you know what you're doing. Even the existing identity flags allow * as a regex, which is basically the same (except it still requires that the SAN and issuer OID be set, which they might not be for private PKI).

How does this sound? Once again, the long-term solution is probably going to be splitting out these commands into opinionated verification and generic verification.

** I am hesitant still to remove the flag in this case, because it's harder to build verification policies around certificates. I treat these flags as policy, but by CLI flag. Ideally these flags would be a part of a richer policy, and we would want to describe verification using identities rather than raw certificates. I also think we still need the flag for bundles, because it is effectively an opaque object to the lay user (yes, they could parse it to find the identity, but they probably won't). It's easier to visualize correctness with an email,

[nsmith5]

I was thinking if a slightly different approach long term, but I think we would both end up at the same place basically.

If you give us a leaf certificate and bundle, we apply the strictest possible interpretation -- we require the exact same leaf certificate to be attached to the OCI image / rekor bundle / offline bundle (from sign-blob --bundle). If your PKI is Fulcio-like or not, we check for all extensions being exactly the same.

If your certificate is just a vehicle for a public key, I feel like you should potentially just pop out the public key with e.g step certificate key cert.pem and verify with that so that there is no interpretation of any x509 extensions at all.

In the near term, we also allow folks to verify without the leaf certificate using the same identity policy flags we have for Fulcio certs (--certificate-identity, --certificate-oidc-issuer, etc etc). This restricts folks to BYO PKI that is Fulcio-like.

Longer term, we namespace the BYO PKI stuff like cosign custom-pki verify and make a really flexible claims checking by passing in something like a Rego policy or other flags?

I feel like this way we don't ever really need to open up a flag like --allow-all-identities, but I'm not sure if I'm missing a use case in this approach

[haydentherapper]

I feel like this way we don't ever really need to open up a flag like --allow-all-identities, but I'm not sure if I'm missing a use case in this approach

Commenting on this first, the rest of my comments are extraneous.
I was thinking about the case where a certificate didn't match fulcio's issued cert profiles, but given your later comment about not supporting that case, I agree we don't need another flag. For the case you gave where you provide identity flags with a certificate, I'm inclined to continue to require those flags. I expect most users will provide a bundle, not only a certificate, and if a bundle is provided, I think requiring identity flags is reasonable. I think this will also be clearer once we use the bundle format to simplify input/output.

We're sorta stuck in this state where we have a lot of ongoing work and a lot of work we want to do. I'd prefer we keep what's here, to always require the flags, and keep an eye on user feedback to see how we can make the UX better.

If you give us a leaf certificate and bundle, we apply the strictest possible interpretation -- we require the exact same leaf certificate to be attached to the OCI image / rekor bundle / offline bundle (from sign-blob --bundle). If your PKI is Fulcio-like or not, we check for all extensions being exactly the same.

At least in verify-blob, the leaf certificate from the bundle will overwrite the provided leaf certificate, see https://github.com/sigstore/cosign/blob/main/cmd/cosign/cli/verify/verify_blob.go#L211. We could change the behavior to verify that both match, but this is probably fine for now. I think it's an unexpected case to pass both, given I think you'd have to hand-craft a bundle to contain only the rekor payload and not the certificate if the bundle is requested on signing.

If your certificate is just a vehicle for a public key, I feel like you should potentially just pop out the public key with e.g step certificate key cert.pem and verify with that so that there is no interpretation of any x509 extensions at all.

I think this is reasonable, though not the stance we've taken currently. I'm totally happy to support this though, it simplifies how we treat certificates, such that we can say that a certificate always means it conforms to fulcio's issued certificate profile. I'd say we proceed with this and if there's any comments from users, we can adapt.

In the near term, we also allow folks to verify without the leaf certificate using the same identity policy flags we have for Fulcio certs (--certificate-identity, --certificate-oidc-issuer, etc etc). This restricts folks to BYO PKI that is Fulcio-like.

Could you clarify this? Do you mean have policy flags for non-fulcio-like certificates? If so, I don't think this is something we should implement. We will have an endlessly growing list of policy flags if so. For example, someone asked to support subject (not SAN) verification, and we proposed relying on an upcoming cosign inspect command to do verification themselves. Supporting something else, like a Rego policy like you suggested, would be a good option.

[nsmith5]

Could you clarify this? Do you mean have policy flags for non-fulcio-like certificates? If so, I don't think this is something we should implement. We will have an endlessly growing list of policy flags if so.

Yeah so the idea here was just to have one more flag like --policy foo.rego so that folks could just pass in a rego policy. Like you said, one off flags would just grow and grow. I feel like should just support arbitrary rego or cue or what ever and then folks can add what ever verification logic they want. This matches the policy-controller support for rego and cue when things just get too complicated and users need an escape hatch for their situation

[haydentherapper]

That sounds like a great feature!

[znewman01]

Great discussion. There's a lot, so trying to summarize:

Immediately

Try to make minimal changes here, in favor of a better long-term solution:

• Disallow the "certificate as a vehicle for a private key". Add an example to the docs using step and cosign import-key.
• Continue requiring identity flags (support BYO PKI only for "Fulcio-like" settings; see Fix verifying when a certificate is provided #2633 ).
• If both a bundle with a certificate, and a certificate are passed, we should require that they match exactly (or just disallow providing both; Make sure a cert passed in via --cert matches the bundle cert #2652 )

Medium-term

• Prioritize cosign inspect ( Add cosign inspect command. #2210 ) as an escape hatch for more complicated checks for now.

Long-term

• Separate CLI verbs for BYO PKI and Keyless will enable a clearer overall UX here.
• Support for a Rego/CUE "escape hatch" to verify commands.
  • (This is actually really nice because we can implement verification by "compiling" baseline verify CLI flags into Rego/CUE.)

---

@haydentherapper @nsmith5 can you make sure I didn't miss anything?

[haydentherapper]

LGTM, thanks @znewman01!

One more immediate AI, a part of minimal changes, is to complete #2633 as a fix for verifying with a provided Fulcio-like certificate.

[avishayil]

Hi, just wondering where it stands.
We have a usecase when we try to sign and verify using an existing x509 certificate and certificate chain. Currently this flow is not working with cosign v2, so we're using cosign v1 for now

[haydentherapper]

Hey @avishayil, I haven't had a chance to fix this. Can you confirm which command you're running that is failing for you?

For OCI, you can also run cosign attach to attach the certificate and chain to the image before running cosign verify, which should workaround the issue.

[chaospuppy]

Just following up on this as it seems to still be an issue with cosign v2.1.1. Additionally, I have not had success using cosign attach to add the certificate and certificate chain.

For background, we are using internal PKI to sign images using a KMS key. In cosign v1.13.1, we had been signing and attaching the certificate using cosign sign --key <kms arn> --certificate cosign-certificate.pem <image>. This signature could then be verified using an offline public key with cosign verify --key cosign-key.pub <image>, and later with v2.x, cosign verify --key cosign-key.pub --insecure-ignore-tlog <image>. However, once our certificate expired and was rotated, verifying images with the old certificate attached would result in expected a signed timestamp to verify an expired certificate errors, which make sense and prompted us to look into implementing Rekor as a method of recording timestamp signatures so this won't happen again in the future.

The following steps should reproduce the (they're essentially the same as above, but to confirm the flow):

Sign the image, attaching the cert and bundle in the process

cosign sign -y \
  --rekor-url='http://rekor.cluster.internal' \
  --key awskms:///<id> 192.168.106.3:5000/rekor-test:latest  \
  --certificate cosign-certificate.pem \
  --certificate-chain cosign-ca-bundle.pem

Attempt to verify image using Rekor

cosign verify \
  --rekor-url http://rekor.cluster.internal \
  --certificate cosign-certificate.pem \
  --certificate-chain cosign-ca-bundle.pem \
  --certificate-identity me@whatever.io \
  --certificate-oidc-issuer-regexp ".*" \
  --insecure-ignore-sct 192.168.106.3:5000/rekor-test:latest

This results in the same error described by the OP:

Error: no matching signatures: error verifying bundle: comparing public key PEMs, expected
-----BEGIN PUBLIC KEY-----
...
, got -----BEGIN CERTIFICATE-----
...

Meanwhile, attempting to cosign attach the bundle and certificates along with the signature using cosign sign --signature-output cosign.sig --upload=false --certificate cosign-certificate.pem --certificate-chain cosign-ca-bundle.pem 192.168.106.3:5000/rekor-test:latest and then cosign attach signature --signature cosign.sig --certificate cosign-certificate.pem --certificate-chain cosign-ca-bundle.pem 192.168.106.3:5000/rekor-test:latest, results in

Error: no matching signatures: error verifying bundle: verifying bundle: rekor log public key not found for payload
 error verifying bundle: verifying bundle: rekor log public key not found for payload
 no valid tlog entries found rekor log public key not found for payload. Check your TUF root (see cosign initialize) or set a custom key with env var SIGSTORE_REKOR_PUBLIC_KEY
main.go:69: error during command execution: no matching signatures: error verifying bundle: verifying bundle: rekor log public key not found for payload
 error verifying bundle: verifying bundle: rekor log public key not found for payload
 no valid tlog entries found rekor log public key not found for payload. Check your TUF root (see cosign initialize) or set a custom key with env var SIGSTORE_REKOR_PUBLIC_KEY

[haydentherapper]

To confirm, does that error occur during cosign attach or during cosign verify? On verification, you'll need to specify --insecure-skip-tlog-verify=true. If this occurred during attachment, we can investigate why as I didn't think verification occurred on upload. It's probably related to expecting a tlog entry.

[chaospuppy]

The last error above occurs during cosign verify. Since we are going through this exercise in order to begin implementing Rekor to resolve our issue with having expired certificates attached to images going forward, I would think --insecure-ignore-tlog would defeat the purpose, of having timestamp signatures in Rekor, correct?

[chaospuppy]

Again the problem we're trying to solve with Rekor is:

• We have self-manage (BYO) PKI
• We have run into a situation where the certificate we were attaching to images along with signatures has expired, which leads to expected a signed timestamp to verify an expired certificate verification errors
• To prevent this from happening again when our next certificate expires, we are looking at using Rekor to keep the transaction logs and provide the signed timestamp so images with expired certs can be verified

NOTE: We have been able to do this using a timestamp service, but self-hosting Rekor is more attractive for better integration/transport of transaction logs to disconnected environments

• Using Rekor in the way described above still results in verification failures, although I'm happy to be told the use is wrong.

[haydentherapper]

Ah, I think I see the issue. You need to also attach the Rekor response ("rekor bundle") to the OCI image to resolve the issue. upload=false is going to prevent any metadata (signature, cert, rekor response) from being uploaded to the container. I would recommend dropping this flag. This should upload all metadata to the container, and you won't need to call cosign attach. Does this work for you?

The workaround using attach was only if you had an existing container and couldn't re-sign it. Unfortunately there is no way to currently attach a Rekor bundle after signing - #2994 is a WIP. Also there is no support to output the Rekor bundle using cosign sign, but I can easily add this if needed though. Would you need this?

Just for background context: To verify short-lived certificates, a Sigstore client needs either:

• A signed timestamp from a timestamp authority whose issuance time occurs during the certificate's validity window
• A persisted response from Rekor ("rekor bundle") whose inclusion time occurs during the certificate's validity window
• The certificate to not be expired, which is typically not the case since the certificate expires in 10 minutes

Cosign also expects a Rekor bundle/response regardless of how the timestamp is obtained, since part of Sigstore's security model is requiring transparency. There's a few issues on GitHub discussing handling private instances better - 1.x had better support, 2.x swung the pendulum a bit too far but made usage of the public instance more secure. insecure-ignore-tlog was added to handle private instances without Rekor, and we have some thoughts on how to improve this in the future (#2736).

[chaospuppy]

I will confirm once I get Rekor stood up again (colima has been sad since Ventura), but I also tried without --upload. Before I send you down a rabbit hole I will double check. We would not need the ability to attach the rekor bundle post-image-push, so that's not an issue.

We did have success with the first option, but wanted to see about the second.

We were utilizing insecure-ignore-tlog once we migrated to v2.0.0 because we did not have our own instance or Rekor and were not keen on using the public Rekor server for the aforementioned transport concerns, but my assumption was that would result in --rekor-url being more or less ignored, and then we would not have timestamp signatures. I am also making the assumption that Rekor creates timestamp signatures at sign time and serves them at verify time, encompassing the role of a timestamp server. Is that correct?

[chaospuppy]

Yeah, confirmed that the issue remains without upload=false:

cosign sign \
  --certificate cosign-certificate.pem \
  --certificate-chain cosign-ca-bundle.pem \
  --rekor-url='http://rekor.cluster.internal' 
  --key awskms:///<id> \
  -y \
  192.168.106.3:5000/rekor:3.4.4

cosign verify \
  --certificate-chain cosign-ca-bundle.pem \
  --certificate cosign-certificate.pem \
  --rekor-url http://rekor.cluster.internal \
  --certificate-identity me@whatever.io \ 
  --certificate-oidc-issuer-regexp ".*" \ 
  --insecure-ignore-sct \
  192.168.106.3:5000/rekor:3.4.4

Throws:

Error: no matching signatures: error verifying bundle: comparing public key PEMs, expected
-----BEGIN PUBLIC KEY-----
...
, got -----BEGIN CERTIFICATE-----
...

[haydentherapper]

You can drop certificate-chain and certificate during cosign verify, since they're already attached to the container and cosign will get them from the container manifest.

I am also making the assumption that Rekor creates timestamp signatures at sign time and serves them at verify time, encompassing the role of a timestamp server. Is that correct?

Rekor acts as a witness to a signing event and will provides a timestamp on upload. If a signer wants to distribute trust to a third-party timestamp authority, it can do so by fetching a timestamp that is signed over the artifact/container signature. The timestamp that comes from either Rekor or the timestamp authority is used during verification.

[chaospuppy]

Without certificate-chain and certificate:

cosign verify --rekor-url http://rekor.cluster.internal --certificate-identity me@whatever.io --certificate-oidc-issuer-regexp ".*" --insecure-ignore-sct 192.168.106.3:5000/rekor:3.4.4
Error: no matching signatures: error verifying bundle: verifying bundle: rekor log public key not found for payload
main.go:69: error during command execution: no matching signatures: error verifying bundle: verifying bundle: rekor log public key not found for payload

Is this a quirk of self-hosted Rekor?

[haydentherapper]

This is because you have to configure Cosign with the roots of trust (Rekor pub key, Fulcio cert chain) from your self-hosted instance. You can either specify trusted roots via CLI flag (SIGSTORE_ROOT_FILE for fulcio, SIGSTORE_REKOR_PUBLIC_KEY for the Rekor public key, SIGSTORE_CT_LOG_PUBLIC_KEY_FILE if you're running a CT log with Fulcio).

By default, Cosign is configured with roots of trust from the public instance that we store in a TUF/TheUpdateFramework repository. You can also set up a TUF repo and initialize Cosign with your own TUF repository. We created a guide for this if you want to pursue this, https://blog.sigstore.dev/sigstore-bring-your-own-stuf-with-tuf-40febfd2badd/, one of the benefits being you can easily update roots of trust and you'll have a setup similar to how we run the public instance of sigstore.

[chaospuppy]

Gotcha, thank you @haydentherapper I'll review the Rekor helm chart.

[dschiemann80]

Hi,

I am hitting the same error as the OP, but in a different scenario:
I use a Yubikey hardware token with an ECDSA signing key for signing with cosign (self-built 2.2.1 with pkcs11 support). I also use the public instance of rekor as transparency log server. Signing works without errors and a tlog is written to rekor.
For verification, I use the exported public key of the ECDSA signing key in the hardware token. When I verify with cosign 2.2.1 release, I get the same error as the OP:

cosign verify --key ecdsa-exported.pub <image>
Error: no matching signatures: error verifying bundle: comparing public key PEMs, expected -----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE++RgJ8QILu2FSgdXoM2ddTnLFPdw5zwk
GodY4zgzmzw2nzRtecxihJOjGZv95L/DYG5kRXNhFj7D+4vPWQpxbgTYVEcm9Oo2
8UhOAWzozUHrfVvZbfK49U6tGGzvl3AN
-----END PUBLIC KEY-----
, got -----BEGIN CERTIFICATE-----
MIIEAzCCA4qgAwIBAgIQDd4PaGRyOb63SaTIDj6aDTAKBggqhkjOPQQDAzBkMQsw

But when I checkout 2.2.1 release as source, cherry-pick the commit from #2633 and build it, the verify works without errors:

./cosign verify --key ecdsa-exported.pub <image>

Verification for <image> --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":...

Is it possible to merge #2633? Or maybe there is another solution in my scenario?

[haydentherapper]

#2633 (comment) is the correct fix for this issue, but this is a larger refactor that I haven't been able to get to. I can try to work on this in the coming weeks, or if anyone wants to pick up #2633 and get it finished and tested, that would be appreciated.