Kubelet: Use RepoDigest for ImageID when available #33014

DirectXMan12 · 2016-09-19T13:55:53Z

Release note:

Use manifest digest (as `docker-pullable://`) as ImageID when available (exposes a canonical, pullable image ID for containers).

Previously, we used the docker config digest (also called "image ID"
by Docker) for the value of the ImageID field in the container status.
This was not particularly useful, since the config manifest is not
what's used to identify the image in a registry, which uses the manifest
digest instead. Docker 1.12+ always populates the RepoDigests field
with the manifest digests, and Docker 1.10 and 1.11 populate it when
images are pulled by digest.

This commit changes ImageID to point to the the manifest digest when
available, using the prefix docker-pullable:// (instead of
docker://)

Related to #32159

This change is

DirectXMan12 · 2016-09-19T13:58:28Z

This is an implementation of the short-term solution I discussed in #32159.

cc @kubernetes/sig-node @derekwaynecarr @ncdc

DirectXMan12 · 2016-09-19T14:00:17Z

cc @dchen1107

derekwaynecarr · 2016-09-19T14:04:32Z

/cc @deads2k

ncdc · 2016-09-19T14:09:57Z

pkg/kubelet/dockertools/docker.go

@@ -153,6 +154,14 @@ func filterHTTPError(err error, image string) error {

 // Check if the inspected image matches what we are looking for
 func matchImageTagOrSHA(inspected dockertypes.ImageInspect, image string) bool {
+	// NB: the code below this check checks for validity of `image` as a pull spec.
+	// However, it can be useful to also inspect images by ID (config digest, e.g.
+	// when determining the manifest digestof the image in use by a container).


s/digestof/digest of/

ncdc · 2016-09-19T14:10:55Z

pkg/kubelet/dockertools/docker.go

@@ -202,6 +211,8 @@ func matchImageTagOrSHA(inspected dockertypes.ImageInspect, image string) bool {
 		}
 		if digest.Digest().Algorithm().String() == id.Algorithm().String() && digest.Digest().Hex() == id.Hex() {
 			return true
+		} else {
+			glog.Infof("DIGEST: %v != %v", digest.Digest().Hex(), id.Hex())


This probably needs to be V(4), and it could be a bit more verbose :-)

Whoops, I think that suck in from debugging.

ncdc · 2016-09-19T14:12:35Z

pkg/kubelet/dockertools/docker.go

@@ -153,6 +154,14 @@ func filterHTTPError(err error, image string) error {

 // Check if the inspected image matches what we are looking for
 func matchImageTagOrSHA(inspected dockertypes.ImageInspect, image string) bool {
+	// NB: the code below this check checks for validity of `image` as a pull spec.
+	// However, it can be useful to also inspect images by ID (config digest, e.g.


I'm not sure I'd ever expect this to happen. You would have to pull the image by tag or digest to a node, then run a pod whose container's image is the value for config.digest, and that pod would have to be scheduled to the same node. Right?

I'm not sure I'm understanding you here, but when you do container-inspect, you get a field called ImageID, which we currently use to populate the ImageID. In order to turn that into a manifest digest, you have to do an image-inspect on that config digest to get the information for the image (which includes the RepoDigests field).

The purpose of this function is to determine if the image parameter matches the inspected image, either by tag or digest. Because there are 3 ways a user can reference an image (tag, digest, id aka config.digest), we have to make sure that the inspected image does not match id/config.digest.

Because inspected.ID is the value of config.digest, the only way for it to equal image is if the user specified the config.digest for containers[i].image in the pod spec. But you can't pull by config.digest, which means that the only way you can successfully run a container whose image is config.digest is if that image already exists on the node. So you would have to:

Run a pod whose image is e.g. foo/bar:latest

Find out what that image's config.digest is

Run another pod, specifying the config.digest value for the image, and make sure the pod is scheduled to the same node from step 1

So, the only way to get into the docker InspectImage API is through the InspectImage method on DockerInterface, which calls this. So the alternative is to add an InspectImageUnchecked to DockerInterface which doesn't verify the inspected info after pulling (or add a check argument to the InspectImage method).

I see what you mean here, though. The intent of the method was not entirely clear based on the godoc above :-/.

ok, switched over to adding a checked argument to InspectImage which optionally skips the check.

I don't like the checked arg added to the function. If the verification is not always needed, why not just move it up to somewhere else? E.g., IsImagePresent might be a good candidate.

ncdc · 2016-09-19T14:12:53Z

pkg/kubelet/dockertools/docker.go

@@ -45,6 +45,7 @@ import (
 const (
 	PodInfraContainerName = leaky.PodInfraContainerName
 	DockerPrefix          = "docker://"
+	DockerPullablePrefix  = "docker-pullable://"


Is there a reason we need a different prefix?

It lets us tell which value (config digest vs manifest digest/spec) we're getting at a glance. Otherwise, there's not an easy way to tell, except by trying to parse and/or inspect the value

I thought we agreed upon introducing a new field called CanonicalImageID to distinguish manifest digest from config digest, and leaving the existing field as is. Why we change that direction?

ncdc · 2016-09-19T14:14:04Z

pkg/kubelet/dockertools/docker_manager.go

+	// default to the image ID, but try and inspect for the RepoDigests
+	imageID := DockerPrefix + iResult.Image
+	imgInspectResult, err := dm.client.InspectImage(iResult.Image)
+	if err != nil {


Do you think this could be cleaner as a switch block?

hmm... like

switch { case err != nil: utilruntime.HandleError(fmt.Errorf("unable to inspect docker image %q while inspecting docker container %q: %v", containerName, iResult.Image, err)) case len(imgInspectResult.RepoDigests) > 1: glog.V(4).Infof("Container %q had more than one associated RepoDigest (%v), only using the first", containerName, imgInspectResult.RepoDigests) fallthrough case len(imgInspectResult.RepoDigests) > 0: imageID = xyz

Maybe? I could also just move the logging if statement out a bit (it doesn't need to be nested like it is). I'm not convinced that a switch is better than that

Yeah, something like that. I was trying to get rid of the nesting if possible.

derekwaynecarr · 2016-09-19T14:31:28Z

FYI @smarterclayton

k8s-bot · 2016-09-19T18:34:26Z

GCE e2e build/test passed for commit 5d3a9669e80d0dbfe33dd7b7c8ed6730ded23426.

simon3z · 2016-09-23T15:15:29Z

Thanks for reusing the imageID field for this, it works well for us (ManageIQ/CloudForms).

Although last time I heard from @deads2k I thought we weren't going to change the prefix (docker:// vs docker-pullable://). Having this information is even better for us so we can actually tell whether we're able to crosslink this with Images or not.

simon3z · 2016-09-23T15:18:06Z

@enoodle @zeari please note that the prefix could change to docker-pullable:// (for your current work on image crosslinking).

derekwaynecarr · 2016-09-23T15:48:13Z

@dchen1107 -- are you able to review this? @DirectXMan12 -- can you make the red x's go away?

dchen1107

I am ok with the change in general, especially it only expose the image ID without changing any behavior for now.

dchen1107 · 2016-09-23T17:29:36Z

pkg/kubelet/dockertools/docker.go

@@ -45,6 +45,7 @@ import (
 const (
 	PodInfraContainerName = leaky.PodInfraContainerName
 	DockerPrefix          = "docker://"
+	DockerPullablePrefix  = "docker-pullable://"


I thought we agreed upon introducing a new field called CanonicalImageID to distinguish manifest digest from config digest, and leaving the existing field as is. Why we change that direction?

DirectXMan12 · 2016-09-26T18:21:37Z

@dchen1107 in the long run, we want to have two separate fields, but we were thinking that we could start here, since we've never explicitly guaranteed the meaning of the field (and you'll be able to distinguish between the old and new meanings using the prefix change).

yujuhong · 2016-09-26T21:40:58Z

@dchen1107 in the long run, we want to have two separate fields, but we were thinking that we could start here, since we've never explicitly guaranteed the meaning of the field (and you'll be able to distinguish between the old and new meanings using the prefix change).

This PR effecitvely changes the meaning of the ImageID field in the container status. I'm not sure if that's acceptable for the users. but assume it is (meaning no users want the config.digest), why would we ever want to change to "two separate fields" in the long run?

vishh · 2016-09-26T22:24:11Z

ImageID was only meant for debugging. Will this change be necessary if the Image in the PodSpec is a digest?

dchen1107 · 2016-09-26T22:55:49Z

@yujuhong This is my concern too. I understand the motivation why @DirectXMan12 want to start from here. But once we build the automated system and logic based on top of this, especially relying on parsing the certain string, it is difficult to change later. My concern might not valid in this case.

@vishh I think there is a need to expose the right / canonical image id, at least at the node level. Given the current situation, before settling down the generic structure to represent different image types, I don't think we can simply introducing digest to PodSpec. What @DirectXMan12 proposed here is a simple and less intrusive way to expose the necessary information, and this change enables building an out-of-band image / package management system at the cluster level for them. What they learnt from that system eventually can benefit OSS kubernetes to define a generic structure for other image types, and build a generic package / image management system at cluster level.

yifan-gu · 2016-09-26T23:05:31Z

cc @euank @jonboulle

dchen1107 · 2016-09-27T17:05:35Z

@DirectXMan12 Can you fix the failing tests?

Previously, we used the docker config digest (also called "image ID" by Docker) for the value of the `ImageID` field in the container status. This was not particularly useful, since the config manifest is not what's used to identify the image in a registry, which uses the manifest digest instead. Docker 1.12+ always populates the RepoDigests field with the manifest digests, and Docker 1.10 and 1.11 populate it when images are pulled by digest. This commit changes `ImageID` to point to the the manifest digest when available, using the prefix `docker-pullable://` (instead of `docker://`)

euank · 2016-10-05T00:43:26Z

@derekwaynecarr My understanding was that we'd add a new one. Sorry for my misunderstanding and getting to this conversation a bit late; my bad!

My concern here is more trying to "keep us honest" to our own standards than practical, and if we want to assert that we don't need to follow the rules for this sort of mostly-useless-field case I'm fine with being overruled.

derekwaynecarr · 2016-10-05T17:55:25Z

@k8s-bot kubemark e2e test this

ncdc · 2016-10-05T18:02:56Z

LGTM

k8s-ci-robot · 2016-10-05T18:36:16Z

Jenkins Kubemark GCE e2e failed for commit 01b0b5e. Full PR test history.

The magic incantation to run this job again is @k8s-bot kubemark e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

derekwaynecarr · 2016-10-05T19:22:16Z

@k8s-bot kubemark e2e test this

derekwaynecarr · 2016-10-05T21:09:08Z

@dchen1107 -- looks like all the check marks are green... ptal

bgrant0607 · 2016-10-07T04:52:07Z

@euank ImageID will always contain docker-pullable://... after this change?

The reason the value in that field contains a prefix is so that we could add other types of image IDs, but we expected to change it upon request only, such as for other kinds of images, like OCI.

@smarterclayton @euank @dchen1107 Why is the current field useless? Feel free to point me at an earlier comment.

euank · 2016-10-07T05:04:43Z

@bgrant0607 It's useless because the current image id it presents, docker://<value>, displays a node-local value which cannot be used to reference the image either on external registries nor other nodes.

This change is about making it so it can whenever possible display a value that is not node-local, but rather a uniquely identifying and reference-able in external repositories. The new value will not always be present iiuc, and the docker:// prefix could still be visible (e.g. on nodes with old docker clients or in the case of other errors).

Since the original spirit of including the prefix was for users to parse and switch on it with the expectation of more, I withdraw my concern.
Thanks for the info, this now LGTM. I also realize that this field already is overloaded to include the rkt:// prefix in that case.

My bad for confusion drawing this out, sorry!

yujuhong · 2016-10-07T15:43:49Z

@euank ImageID will always contain docker-pullable://... after this change?

After this change, if an image digest is available, kubele will report docker-pullable://. Otherwise, it'd report docker://. The digest may not be available if you pulled by name:tag originally.

derekwaynecarr · 2016-10-07T17:36:33Z

@bgrant0607 -- it sounds like there are no more remaining concerns? Objections to merging?

bgrant0607 · 2016-10-07T18:01:25Z

@derekwaynecarr I discussed the change with @yujuhong and it's ok with me from an API-compatibility point of view. She'll make a final pass over the PR.

yujuhong · 2016-10-07T18:24:10Z

LGTM.

k8s-github-robot · 2016-10-07T19:05:44Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2016-10-07T19:48:32Z

Automatic merge from submit-queue

wojtek-t · 2016-10-08T08:19:18Z

This PR broke all kubemark suites. I'm reverting it.

@yujuhong

Automatic merge from submit-queue CRI: Image pullable support in dockershim For #33189. The new test `ImageID should be set to the manifest digest (from RepoDigests) when available` introduced in #33014 is failing, because: 1) `docker-pullable://` conversion is not supported in dockershim; 2) `kuberuntime` and `dockershim` is using `ListImages with image name filter` to check whether image presents. However, `ListImages` doesn't support filter with `digest`. This PR: 1) Change `kuberuntime.IsImagePresent` to use `runtime.ImageStatus` and `dockershim.InspectImage` instead. ***Notice an API change: `ImageStatus` should return `(nil, nil)` for non-existing image.*** 2) Add `docker-pullable://` support. 3) Fix `RemoveImage` in dockershim #29316. I've tried myself, the test can pass now. @yujuhong @feiskyer @yifan-gu /cc @kubernetes/sig-node

googlebot added the cla: yes label Sep 19, 2016

DirectXMan12 force-pushed the feature/set-image-id-manifest-digest branch from d374914 to ffee60b Compare September 19, 2016 13:56

ncdc suggested changes Sep 19, 2016

View reviewed changes

derekwaynecarr added this to the v1.5 milestone Sep 19, 2016

DirectXMan12 force-pushed the feature/set-image-id-manifest-digest branch 2 times, most recently from d7a291f to 801f385 Compare September 19, 2016 14:42

k8s-github-robot assigned dchen1107 Sep 19, 2016

k8s-github-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. release-note-label-needed labels Sep 19, 2016

DirectXMan12 force-pushed the feature/set-image-id-manifest-digest branch from 801f385 to 5d3a966 Compare September 19, 2016 17:56

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Sep 19, 2016

derekwaynecarr added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-label-needed labels Sep 19, 2016

dchen1107 reviewed Sep 24, 2016

View reviewed changes

enoodle mentioned this pull request Sep 26, 2016

Container Images: Collect container images from Openshift ManageIQ/manageiq#10692

Closed

k8s-github-robot mentioned this pull request Oct 5, 2016

[k8s.io] Networking should check kube-proxy urls {Kubernetes e2e suite} #32436

Closed

ncdc approved these changes Oct 5, 2016

View reviewed changes

ncdc mentioned this pull request Oct 6, 2016

Docker image reported by status is not the image digest #14047

Closed

bgrant0607 assigned bgrant0607 and unassigned dchen1107 Oct 7, 2016

bgrant0607 assigned yujuhong and unassigned bgrant0607 Oct 7, 2016

yujuhong added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 7, 2016

k8s-github-robot merged commit c23346f into kubernetes:master Oct 7, 2016

Random-Liu mentioned this pull request Oct 8, 2016

CRI: Image pullable support in dockershim #34380

Merged

wojtek-t mentioned this pull request Oct 8, 2016

Revert "Kubelet: Use RepoDigest for ImageID when available" #34386

Merged

DirectXMan12 mentioned this pull request Oct 10, 2016

Kubelet: Use RepoDigest for ImageID when available #34473

Merged

ezimanyi mentioned this pull request Oct 29, 2020

Pod status.containerStatuses[].imageId depends on container runtime #95968

Closed

Kubelet: Use RepoDigest for ImageID when available #33014

Kubelet: Use RepoDigest for ImageID when available #33014

Conversation

DirectXMan12 commented Sep 19, 2016 • edited by k8s-oncall Loading

DirectXMan12 commented Sep 19, 2016

DirectXMan12 commented Sep 19, 2016

derekwaynecarr commented Sep 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ncdc Sep 19, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

derekwaynecarr commented Sep 19, 2016

k8s-bot commented Sep 19, 2016

simon3z commented Sep 23, 2016 • edited Loading

simon3z commented Sep 23, 2016

derekwaynecarr commented Sep 23, 2016

dchen1107 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DirectXMan12 commented Sep 26, 2016

yujuhong commented Sep 26, 2016

vishh commented Sep 26, 2016

dchen1107 commented Sep 26, 2016

yifan-gu commented Sep 26, 2016

dchen1107 commented Sep 27, 2016

euank commented Oct 5, 2016

derekwaynecarr commented Oct 5, 2016

ncdc commented Oct 5, 2016

k8s-ci-robot commented Oct 5, 2016

derekwaynecarr commented Oct 5, 2016

derekwaynecarr commented Oct 5, 2016

bgrant0607 commented Oct 7, 2016

euank commented Oct 7, 2016 • edited Loading

yujuhong commented Oct 7, 2016

derekwaynecarr commented Oct 7, 2016

bgrant0607 commented Oct 7, 2016

yujuhong commented Oct 7, 2016

k8s-github-robot commented Oct 7, 2016

k8s-github-robot commented Oct 7, 2016

wojtek-t commented Oct 8, 2016

DirectXMan12 commented Sep 19, 2016 •

edited by k8s-oncall

Loading

ncdc Sep 19, 2016 •

edited

Loading

simon3z commented Sep 23, 2016 •

edited

Loading

euank commented Oct 7, 2016 •

edited

Loading