Digest validation on Install #355

jeremyrickard · 2018-11-05T18:47:38Z

Validate docker image digest before doing a Docker pull on duffle install

Updates action/driver code to pass the Digest along with image so it can be validated.

Closes: #233

jeremyrickard · 2018-11-05T18:53:53Z

@simonferquel Per #233, we wish to validate the digest matches what is in the bundle.json for a given invocation image.

Is the first commit sufficient (i.e. just perform a DistributionInspect) or should we get the manifest and recompute the digest to verify (second commit, using the containerD libraries)?

technosophos · 2018-11-05T19:56:21Z

I'm not sure I understand this PR. The comment says that we want to validate against the bundle.json, but I don't see that here. And it doesn't appear that op.ImageDigest ever gets set in this PR.

jeremyrickard · 2018-11-05T20:08:02Z

pkg/action/action.go

@@ -71,6 +71,7 @@ func opFromClaim(action string, c *claim.Claim, ii bundle.InvocationImage, creds
 		Parameters:   c.Parameters,
 		Image:        ii.Image,
 		ImageType:    ii.ImageType,
+		ImageDigest:  ii.Digest,


@technosophos if the digest is present for the invocation image in the bundle.json (as https://github.com/deislabs/cnab-spec/blob/master/101-bundle-json.md indicates is should be) , won't it be set here?

opFromClaim gets invoked on the install command AFAICT. I modified the hello helm bundle.json to include the digest (none of them seem to have that in the samples repo now). With this change, that ends up in the docker_driver.go

jeremyrickard · 2018-11-05T20:10:03Z

@technosophos I tried to comment in line where I believe that happens (opFromClaim..). Did I misread that part of the spec:

The digest field must contain a digest, in OCI format, to be used to compute the integrity of the image. The calculation of how the image matches the digest is dependent upon image type. (OCI, for example, uses a Merkle tree while VM images are checksums.)

So in theory, it should be there and when the invocation image config is loaded from from the bundle.json, it should be present (none of our examples do that currently though).

I'm working on adding the logic to fetch the actual blobs to verify things right now, I should have put a WIP on the title. (oops I did and accidentally removed it)

technosophos · 2018-11-05T20:15:10Z

Oh! Yes, your read is correct. In bundle.json, both the invocation images and the regular images should have digests attached to them. And since only one invocation image is executed per install, then there would be one relevant digest for an action/claim.

But, yeah, without the WIP, I thought this was all done and couldn't figure out what I was reviewing. Sorry.

simonferquel · 2018-11-06T09:05:47Z

@jeremyrickard actually I have no idea about this one :(.
I think a simpler solution would be to change a bit the logic behind build and push:

On build, generate an image tag consisting of the hash of the build context + build settings used as the image receipe (I think we only hash the build context right now)
On push,
- first, push the image with this tag on the registry an retrieve the digested reference from the registry (myimage: reference would become myimage@sha256:)
- patch the local bundle with the digested reference, re-sign it if it was signed
- push the bundle
  This way, the system would rely on un-modifiable digested-reference, and would not need to do anything locally to validate the image content.

Also note that if the image has already been pushed by the user using docker CLI, doing an inspect call on that image should report the digested reference in addition to the tagged reference (and so we can skip the "push image phase"). Also if the local bundle has a digested reference, we also can skip all this and directly push the bundle

bacongobbler · 2018-11-06T16:30:23Z

On build, generate an image tag consisting of the hash of the build context + build settings

From discussions last week with @itowlson and @radu-matei , I believe we're planning on ripping that image tag generation code in favour of a version field in duffle.json. The image tag for the bundle is supplied from the cnab component's shasum of the build context, which isn't at all intuitive to the user.

EDIT: see #361 for follow-up discussion on that topic

As far as the push proposal, that sounds like how we'd want to push a "thick" bundle to a registry, right? As far as I understand we may not support pushing a thick bundle to a registry, as the use cases where a thick bundle is required are limited in scope to "export to a USB stick and run it on an IoT device". @technosophos did we get any feedback on whether an exported bundle should be push-able to a registry?

That being said, if we do intend to support pushing thick bundles, I'm a little concerned about patching and re-signing a bundle without the user being able to weigh in on whether the bundle should be patched/re-signed. I like the idea to rely on digests, but duffle export should be able to handle that use case since it'll write out a "thick" bundle.json. I wouldn't want my bundles to be patched on-the-fly with no user intervention on duffle push. Is there a particular reason why we'd want to live-patch image references on duffle push? Is that something docker does for image manifests?

jeremyrickard · 2018-11-06T17:04:40Z

I've amended this so that it has a single commit now.

Summary:

new pkg/digest (request a validator based on image type)
implementation of docker validation in there using containerD libraries
updated pkg/action/install.go to validate digests, if they are present in the bundle.json.

I opened a PR in the cnab-spec repo to make changes to the images list specification to make it similar to invocationImage (notably, adding an imageType). Right now, this PR defaults to just docker.

Closes: #233

jeremyrickard · 2018-11-06T19:09:22Z

pkg/digest/docker_digester.go

+	h := remotes.FetchHandler(cs, fetcher)
+	// This traverses the OCI descriptor to fetch the image and store it into the local store initialized above.
+	// All content hashes are verified in this step
+	if err := images.Dispatch(ctx, h, desc); err != nil {


For anyone wondering how this validates the digest....

This ends up fetching the descriptor from the registry, then fetching the blob associated with that. then it does the same for children layers. When they are written to the temporary file storage, the hash is calculated for each blob. If this is successful (this is via the fetch handler). It's essentially doing a pull, which then gets deleted.

michelleN · 2018-11-08T23:17:42Z

pkg/action/install.go

+		return fmt.Errorf("unable to get image validator: %s", err)
+	}
+	ctx := context.Background()
+	if invocImage.Digest != "" {


What should the expected default behavior be if no digest is provided for the invocation image? From this line, it looks like if there is no digest, we do not validate.

I think we should require a digest and emit an error by default and for when someone is developing a bundle, we should override that default behavior with a flag. I'll have to check if the spec says anything about requiring a digest.

It is currently skipping if there is no digest in the bundle. I agree that we probably should enforce it being present, I think the spec said it must be there and we haven’t enforced that. Any suggestion on what the flag should be?

Also I meant to leave a comment saying we should fix the bundles up and then fix this, but I forgot :(

technosophos · 2018-11-16T15:25:15Z

Needs a quick rebase

jeremyrickard · 2018-11-16T23:16:54Z

Updated the PR to.....

use the updated api stuff to handle authenticated apis
check the digest for local images against what the bundle expects to have without pulling

michelleN · 2019-04-30T16:53:35Z

Should we go ahead and close this one out @jeremyrickard ?

jeremyrickard changed the title ~~WIP: Digest validation on Install~~ Digest validation on Install Nov 5, 2018

jeremyrickard commented Nov 5, 2018

View reviewed changes

jeremyrickard changed the title ~~Digest validation on Install~~ WIP: Digest validation on Install Nov 5, 2018

technosophos added the WIP work in progress label Nov 5, 2018

jeremyrickard force-pushed the digest-validation branch from 81d1776 to 9e232d7 Compare November 6, 2018 15:44

jeremyrickard changed the title ~~WIP: Digest validation on Install~~ Digest validation on Install Nov 6, 2018

jeremyrickard removed the WIP work in progress label Nov 6, 2018

Digest validation on install action

0c9f2eb

Closes: #233

jeremyrickard force-pushed the digest-validation branch from 9e232d7 to 0c9f2eb Compare November 6, 2018 17:47

jeremyrickard commented Nov 6, 2018

View reviewed changes

michelleN reviewed Nov 8, 2018

View reviewed changes

This was referenced Nov 9, 2018

Add flag to duffle install to disable digest validation #391

Open

Make digest validation work with local images. #393

Closed

michelleN mentioned this pull request Nov 13, 2018

duffle export should validate invocation image digests #408

Open

jeremyrickard added 4 commits November 16, 2018 08:50

Merge remote-tracking branch 'upstream/master' into digest-validation

ef921da

Update to use the Docker CLI

cab4e22

Merge remote-tracking branch 'upstream/master' into digest-validation

706eb37

Handle local image verification

f023541

Fixing lint errors

203f0ef

jeremyrickard closed this Jun 21, 2019

jeremyrickard mentioned this pull request Aug 12, 2019

Enforce digest validation cnabio/cnab-go#101

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Digest validation on Install #355

Digest validation on Install #355

jeremyrickard commented Nov 5, 2018 •

edited

Loading

jeremyrickard commented Nov 5, 2018 •

edited

Loading

technosophos commented Nov 5, 2018

jeremyrickard Nov 5, 2018

jeremyrickard commented Nov 5, 2018 •

edited

Loading

technosophos commented Nov 5, 2018

simonferquel commented Nov 6, 2018

bacongobbler commented Nov 6, 2018 •

edited

Loading

jeremyrickard commented Nov 6, 2018

jeremyrickard Nov 6, 2018

michelleN Nov 8, 2018

jeremyrickard Nov 9, 2018

jeremyrickard Nov 9, 2018

technosophos commented Nov 16, 2018

jeremyrickard commented Nov 16, 2018 •

edited

Loading

michelleN commented Apr 30, 2019

Digest validation on Install #355

Digest validation on Install #355

Conversation

jeremyrickard commented Nov 5, 2018 • edited Loading

jeremyrickard commented Nov 5, 2018 • edited Loading

technosophos commented Nov 5, 2018

jeremyrickard Nov 5, 2018

Choose a reason for hiding this comment

jeremyrickard commented Nov 5, 2018 • edited Loading

technosophos commented Nov 5, 2018

simonferquel commented Nov 6, 2018

bacongobbler commented Nov 6, 2018 • edited Loading

jeremyrickard commented Nov 6, 2018

jeremyrickard Nov 6, 2018

Choose a reason for hiding this comment

michelleN Nov 8, 2018

Choose a reason for hiding this comment

jeremyrickard Nov 9, 2018

Choose a reason for hiding this comment

jeremyrickard Nov 9, 2018

Choose a reason for hiding this comment

technosophos commented Nov 16, 2018

jeremyrickard commented Nov 16, 2018 • edited Loading

michelleN commented Apr 30, 2019

jeremyrickard commented Nov 5, 2018 •

edited

Loading

jeremyrickard commented Nov 5, 2018 •

edited

Loading

jeremyrickard commented Nov 5, 2018 •

edited

Loading

bacongobbler commented Nov 6, 2018 •

edited

Loading

jeremyrickard commented Nov 16, 2018 •

edited

Loading