Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add layout package for writing and loading signatures from disk #1040

Merged
merged 8 commits into from
Nov 19, 2021

Conversation

priyawadhwa
Copy link
Contributor

@priyawadhwa priyawadhwa commented Nov 12, 2021

Added a layout package to pkg/oci which should make it easy to store images & associated signatures to disk! This is the first part of implementing #1015

TODOs:

  • unit tests
  • moving the sigLayer implementation into a separate package since it's used by remote as well

I'm still not sure if I implemented this the correct way, cc'ing @jonjohnsonjr and @mattmoor in case I'm totally on the wrong track 😅

Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
Copy link
Member

@mattmoor mattmoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely a good start. Very cool (and unexpected!) to see 🤩

pkg/oci/siglayer/siglayer.go Outdated Show resolved Hide resolved
pkg/oci/siglayer/siglayer.go Outdated Show resolved Hide resolved
pkg/oci/siglayer/siglayer.go Outdated Show resolved Hide resolved
pkg/oci/layout/image.go Outdated Show resolved Hide resolved
pkg/oci/layout/image.go Outdated Show resolved Hide resolved
pkg/oci/layout/write.go Outdated Show resolved Hide resolved
pkg/oci/layout/write.go Outdated Show resolved Hide resolved
pkg/oci/layout/write.go Outdated Show resolved Hide resolved
pkg/oci/layout/image.go Outdated Show resolved Hide resolved

// SignedImage provides access to a remote image reference, and its signatures.
func SignedImage(path string) (oci.SignedImage, error) {
p, err := layout.FromPath(imagePath(path))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this ~always an index? If so, I'm wondering if we should only expose SignedIndex 🤔 (or at least start there instead, and add "sugar" later)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The index is the entrypoint into this layout -- you could interpret the entire layout as a single index and push that to a registry, but I think it's more interesting to just use this index data structure as a list of what's contained within the directory. You can have a bunch of cosign-isms around annotations on the top-level index.json index that are meaningful to cosign without them leaking from this implementation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be interesting to have an idea of a "snapshot" of a SignedImage, that is implemented as an index that points to every piece of relevant metadata?

Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
Priya Wadhwa added 2 commits November 15, 2021 17:17
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
@priyawadhwa priyawadhwa changed the title WIP: Add layout package for writing and loading signatures from disk Add layout package for writing and loading signatures from disk Nov 15, 2021
Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
@priyawadhwa
Copy link
Contributor Author

Hey @jonjohnsonjr & @mattmoor, I refactored a bit to store everything in a single index as recommended. The sample index.json looks like this right now, with a manifest for the image and a manifest for the signatures (distinguishable by annotation). Let me know what you think :)

{
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 528,
         "digest": "sha256:69704ef328d05a9f806b6b8502915e6a0a4faa4d72018dc42343f511490daf8a",
         "annotations": {
            "dev.cosignproject.cosign/image": "true"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 501,
         "digest": "sha256:4c4dc104ee4357651d17a2e4f8faf2d64457acc7fff2d306312abf69c7bbe4c9",
         "annotations": {
            "dev.cosignproject.cosign/sigs": "true"
         }
      }
   ]
}

Comment on lines +28 to +29
imageAnnotation = "dev.cosignproject.cosign/image"
sigsAnnotation = "dev.cosignproject.cosign/sigs"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlorenc What's standard here? should this be dev.sigstore.cosign? Are we documenting these anywhere?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably should be dev.sigstore but I apparently never switched them over from the first version, so dev.cosignproject is currently everywhere. We should open a separate issue to track/document/change all of these.

pkg/oci/layout/index.go Outdated Show resolved Hide resolved
pkg/oci/layout/index.go Outdated Show resolved Hide resolved
sigs, err := si.Signatures()
if err != nil {
return errors.Wrap(err, "getting signatures")
}
if err := write(path, signaturesPath, sigs); err != nil {
return errors.Wrap(err, "writing signatures")
if err := appendImage(layoutPath, sigs, sigsAnnotation); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the resulting index were pushed, how would it run?

cc @jonjohnsonjr

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally clients take the first entry that matches their criteria (platform).

This might be a reasonable knob to expose when pushing to a registry? By default, just push all the entries in index.json, but optionally include that index.json as a top-level index for everything related to the artifact. I don't know how this is intended to be used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for cosign load I was planning on pushing everything in index.json for now and not including the index.json itself so this effectively is the same as running cosign copy -- i think it would be cool to optionally include pushing index.json as a feature for later (although running into some errors with it atm that would need to be addressed)

…gnedImage

Signed-off-by: Priya Wadhwa <priyawadhwa@google.com>
Copy link
Member

@dlorenc dlorenc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks close enough! Merging so we can keep going!

@dlorenc dlorenc merged commit f8f0f6d into sigstore:main Nov 19, 2021
@github-actions github-actions bot added this to the v1.4.0 milestone Nov 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants