Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define a new sigstore Bundle type to include additional verification data and materials #2422

Closed
wants to merge 2 commits into from

Conversation

hectorj2f
Copy link
Contributor

Summary

As part of this effort #2331, I'm proposing renaming the original RekorBundle type to a more generic sigstore Bundle type that includes additional data. The new fields added to the sigstore Bundle are inspired from the protobuf specification here: https://github.com/sigstore/protobuf-specs/blob/main/protos/sigstore_bundle.proto.

The next actions include adding support to use a Timestamp Authority server (sigstore/timestamp-authority) to guarantee a signature happened during the cert's validity period.

Release Note

Documentation

Signed-off-by: Hector Fernandez <hector@chainguard.dev>
Signed-off-by: Hector Fernandez <hector@chainguard.dev>
// RekorPayload holds metadata about recording a Signature's ephemeral key to
// a Rekor transparency log.
// Use Payload instead of TransparencyLogEntry to keep backwards compatibility.
Payload RekorPayload
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept the Payload field instead of renaming it to TransparencyLogEntry, as specified in the protobuf spec, to keep backwards compatibility.

@@ -87,7 +87,7 @@ func TestOptions(t *testing.T) {
LayerMediaType: ctypes.SimpleSigningMediaType,
ConfigMediaType: types.OCIConfigJSON,
Annotations: map[string]string{
BundleAnnotationKey: "{\"SignedEntryTimestamp\":null,\"Payload\":{\"body\":null,\"integratedTime\":0,\"logIndex\":0,\"logID\":\"\"}}",
BundleAnnotationKey: "{\"Payload\":{\"body\":null,\"integratedTime\":0,\"logIndex\":0,\"logID\":\"\"},\"SignedEntryTimestamp\":null,\"EntryTimestampAuthority\":null,\"CertBytes\":null,\"PublicKeyIdentifier\":\"\"}",
Copy link
Contributor Author

@hectorj2f hectorj2f Nov 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the main difference with the new bundle type. Whenever we start using a TSA the bundle will look more like:

"Bundle":{"Payload":{"body":null,"integratedTime":0,"logIndex":0,"logID":""},"SignedEntryTimestamp":null,"EntryTimestampAuthority":"MIIEbjADAgEAMIIEZQYJKoZIhvcNAQcCoIIEVjCCBFICAQExDTALBglghkgBZQMEAgEwgdQGCyqGSIb3DQEJEAEEoIHEBIHBMIG+AgEBBgkrBgEEAYO/MAIwMTANBglghkgBZQMEAgEFAAQgLFsLP5xZKtHz5LM1jdA+K9GQFwI20N5BnnKOOgsPpvcCFQDKNZaXbkFPJh0AD1YsJ27y+SllNxgTMjAyMjExMDcxNzEzNDQrMDEwMDADAgEBAhReYQUVtj8oPw/AXX8xVrnIZfq6Y6A0pDIwMDEOMAwGA1UEChMFbG9jYWwxHjAcBgNVBAMTFVRlc3QgVFNBIFRpbWVzdGFtcGluZ6CCAckwggHFMIIBaqADAgECAhRHCu9dHKS97mFo1cH5neJubRibujAKBggqhkjOPQQDAjAoMQ4wDAYDVQQKEwVsb2NhbDEWMBQGA1UEAxMNVGVzdCBUU0EgUm9vdDAeFw0yMjExMDMxMTUzMThaFw0zMTExMDMxMTU2MThaMDAxDjAMBgNVBAoTBWxvY2FsMR4wHAYDVQQDExVUZXN0IFRTQSBUaW1lc3RhbXBpbmcwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATo3W6NQrpx5D8z5IvgD2DlAgoJMF4KPY9Pj4UfFhfOq029ryszXp3460Z7N+x86bDvyjVrHaeiPnl1HO9Q52zso2owaDAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0OBBYEFHSIhDdTGIsodML/iUOhx7hgo/K7MB8GA1UdIwQYMBaAFBoZYijuouZCvKDtBd0eCyaU2HWoMBYGA1UdJQEB/wQMMAoGCCsGAQUFBwMIMAoGCCqGSM49BAMCA0kAMEYCIQCmPVr5kwYe4Jg9PGO6apgfzSrKAtESgNHpAbE3iIvJhQIhAJIGNxshJcC8LXHRrVWM77no3d3GguSvR01OAPZwE2pqMYIBmDCCAZQCAQEwQDAoMQ4wDAYDVQQKEwVsb2NhbDEWMBQGA1UEAxMNVGVzdCBUU0EgUm9vdAIURwrvXRykve5haNXB+Z3ibm0Ym7owCwYJYIZIAWUDBAIBoIHqMBoGCSqGSIb3DQEJAzENBgsqhkiG9w0BCRABBDAcBgkqhkiG9w0BCQUxDxcNMjIxMTA3MTYxMzQ0WjAvBgkqhkiG9w0BCQQxIgQgGQ4kDSvQ8UJgZaIIKWZg9R0L06St8OBAYrj2VGSq7rcwfQYLKoZIhvcNAQkQAi8xbjBsMGowaAQgXqxJD0nAgg6en9P1bRrU7+6tzxOMn3YThreg7uR6T7EwRDAspCowKDEOMAwGA1UEChMFbG9jYWwxFjAUBgNVBAMTDVRlc3QgVFNBIFJvb3QCFEcK710cpL3uYWjVwfmd4m5tGJu6MAoGCCqGSM49BAMCBEcwRQIhALiLTG7glyyiiHJ33DsWIi24rHpcSnW/EeLfNqaXty4VAiBG7moX2xoOwOfJm8DZveTox9tpcV6NnIN/3LzGs8eSgA==","CertBytes":null, "PublicKeyIdentifier": ""}

// a Rekor transparency log.
// Use Payload instead of TransparencyLogEntry to keep backwards compatibility.
Payload RekorPayload

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am awaiting feedback before adding other fields such as the messageSignature included in the spec.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those fields overlap with LocalSignedPayload too. The top level fields in that would ideally be removed, or else it would open mismatch issues.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! That could easily happen.

@hectorj2f
Copy link
Contributor Author

@asraa @priyawadhwa @haydentherapper I'd like to hear your thoughts about this proposal 👀 . I decided to adapt the current implementation to the fields in the protobuf spec. Also, I included a new field EntryTimestampAuthority that will contains the response as received from the TSA server.

@haydentherapper
Copy link
Contributor

If we are beginning to adopt the Sigstore bundle format, I would prefer we use exactly what has been discussed and approved. Otherwise we now have a third format effectively. If we want to deviate, we should discuss the change in the protobuf specs repo first imo

cc @kommendorkapten

Copy link
Contributor

@priyawadhwa priyawadhwa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks really good! just one question.

)

type Bundle struct {
VerificationData
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference between VerificationData and VerificationMaterial? I wonder if we could come up with more descriptive names so it's more clear :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to port that comment to the upstream bundle, but these are terms being used by those protos.
Data is extra auxiliary data for verification (timestamps, rekor entries), but verification material is the direct material (key hint, certificates)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am gonna port these and other questions to the upstream bundle. Thanks.

PublicKeyIdentifier string
}

// TimestampVerificationData contains various timestamped data following RFC3161.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed in the protobuf-specs repo, SETs are not just timestamp verification data, hence why we went with this nested format - https://github.com/sigstore/protobuf-specs/blob/main/protos/sigstore_bundle.proto#L53-L59

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created one issue here sigstore/protobuf-specs#13. Hopefully we can get more ppl involved on this discussion.

SignedEntryTimestamp []byte

// EntryTimestampAuthority contains the recorded entry from timestamp authority server.
EntryTimestampAuthority []byte
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer a name that includes RFC3161 rather than Timestamp Authority, as the former makes it clear what the format of the time record is

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it makes sense. I like it.

EntryTimestampAuthority []byte
}

func EntryToBundle(tLogEntry *models.LogEntryAnon, signedEntryTimestamp, entryTimestampAuthority, certBytes []byte, pubKeyID string) *Bundle {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we need tests but I wanted to firstly get some feedback, as I expected things might change.

@priyawadhwa
Copy link
Contributor

@haydentherapper this isn't a third format, it's just an addition to the current format. Definitely agree we should also plan for it in the protobuf work, but IIUC that work isn't ready to be used yet (@kommendorkapten please correct me if I'm wrong!). I think it's ok to amend the current format so that we can start testing out this feature & then switch over to the protobufs whenever they're ready.

@@ -62,7 +62,7 @@ func uploadToTlog(ctx context.Context, sv *sign.SignerVerifier, rekorURL string,
return nil, err
}
fmt.Fprintln(os.Stderr, "tlog entry created with index:", *entry.LogIndex)
return cbundle.EntryToBundle(entry), nil
return cbundle.EntryToBundle(entry, []byte{}, []byte{}, []byte{}, ""), nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use nil instead of []byte{}?

)

type Bundle struct {
VerificationData
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to port that comment to the upstream bundle, but these are terms being used by those protos.
Data is extra auxiliary data for verification (timestamps, rekor entries), but verification material is the direct material (key hint, certificates)

// a Rekor transparency log.
// Use Payload instead of TransparencyLogEntry to keep backwards compatibility.
Payload RekorPayload

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those fields overlap with LocalSignedPayload too. The top level fields in that would ideally be removed, or else it would open mismatch issues.

@haydentherapper
Copy link
Contributor

I think we can begin to adopt the format, lemme know otherwise @kommendorkapten. I'm worried about introducing an internal Bundle structure, as it's going to be harder to differentiate between the new bundle format, this bundle structure, and what Rekor returns. We also had a lot of discussion about what the role of the timestamp is when designing the protobuf and designing it to be future proof, and I think those same issues are popping up here.

If we don't want to adopt the new bundle format yet, I'd prefer we make a struct that makes it clear that it's not another bundle format, but still reuse as much as possible from the bundle format wrt to timestamp fields

b := &Bundle{}
// If none of the verification data is configured then return nil
if (tLogEntry == nil || tLogEntry.Verification == nil) && len(entryTimestampAuthority) == 0 {
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why return nil over returning an empty bundle? id be worried about accidental dereferences later

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with you but tLogEntry == nil || tLogEntry.Verification == nil is reusing the old behavior that also returned a nil bundle.

@znewman01
Copy link
Contributor

I think we can begin to adopt the format, lemme know otherwise @kommendorkapten. I'm worried about introducing an internal Bundle structure, as it's going to be harder to differentiate between the new bundle format, this bundle structure, and what Rekor returns. We also had a lot of discussion about what the role of the timestamp is when designing the protobuf and designing it to be future proof, and I think those same issues are popping up here.

I broadly agree. That said, I don't expect client implementations to keep the protobuf representations throughout the program; rather, I'd expect them to "parse" the protobuf out into language-idiomatic objects that are a little bit smarter than the generated proto code. This helps us reject invalid representations that are valid in protobuf (e.g., invalid DER).

Here's what I'd propose:

  • We will want a "bundle" type in Golang eventually, so this can become that.
  • Until the protobuf-specs bundle is a little more mature, this PR should remain internal-only which will allow refactoring once the bundle is stabilized.
  • However, any deviations from the protobuf spec should be (1) heavily commented, and (2) justified with language-specific reasoning. If there's a non-language-specific objection, let's get it fixed upstream.

Copy link
Contributor

@znewman01 znewman01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very-much breaking change to the Golang API. Can we make it official that this API isn't supported by any versioning guarantees?

See also #2365 (reply in thread)

// signatures or any additional timestamped verifications.
type VerificationMaterial struct {
// A chain of X.509 certificates.
CertBytes []byte
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd expect this to be a crypto/x509.Certificate

I guess really...is this struct a data format or an object? My preference is always to move parsing as early as possible so we don't have to handle parse errors in the middle of the program logic.

@kommendorkapten
Copy link
Member

The Sigstore bundle from https://github.com/sigstore/protobuf-specs is in a decent shape I would say. Is it worthy to cut a release in the current stage? Probably no, but I don't think we are miles away. For the npm build provenance we are working on two different implementations of the bundle, generation and verification. One in Go, and one in JavaScript. So we are getting real world testing of the usability now. My feeling is that we will see some small changes in the imminent future, like sigstore/protobuf-specs#10, and cut a release within a few weeks. All the work on the bundle we are doing is expected to be part of sigstore-js and sigstore-go.

Beyond the first release, there are a few expected breaking changes, but the horizon for them is a bit unclear, as we are waiting for e.g. Rekor SET V2.
The bundle is versioned, so it shouldn't be an issue for clients to support multiple versions. If cosign waits until the next bundle release for adoption, I think that can be reasonable too, as long as the this PR doesn't stray away too much and makes the adoption harder.

@hectorj2f
Copy link
Contributor Author

Thanks everyone for the review and suggestions. I'll keep this PR open so I can take over on these changes whenever there aren't major breaking changes expected from the protobuf spec.

In the meantime, I'll try to move on with #2331, and open a different PR with a smaller change that just renames the struct name from RekorBundle to Bundle, and adds a new field EntryRFC3161 to record the timestamped entry from sigstore/timestamp-authority.

@kommendorkapten
Copy link
Member

If it's of any help, we are ready to merge this PR sigstore/protobuf-specs#12 which adds the generated Go code to the repository, so you don't need to rewrite any structs -- you can import them directly from that repository.

@hectorj2f
Copy link
Contributor Author

Closing this for now! I'll re-open it when starting this work again.

@hectorj2f hectorj2f closed this Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants