Add support for referring to images by digest #11109

Merged
merged 1 commit into from Mar 17, 2015

Projects

None yet
@ncdc
Contributor
ncdc commented Mar 2, 2015

Overview

When I create a container, I may specify an image such as mysql:latest. When the image is pulled, latest is resolved to a particular image at that point in time. If I later want to add more containers (e.g. possible read slaves in the MySQL case), all the new containers must use the exact same image as my first container. Using a tag isn't sufficient as the tag can be updated to point at a different image. We need a way to refer to images using immutable identifiers.

With the v2 image format, image manifests have content-addressable, immutable digests. The v2 registry supports retrieving manifests by digest; this pull request adds the corresponding support to the Docker Core.

Digest reference format

We'll need to provide a means to reference an image by its digest. I'd like to propose the following format:

namespace/repository@digest

Supported commands

We'll need to make sure the following commands continue to work as they currently do, as well as with an optional digest:

  • docker build
  • docker pull
  • docker create
  • docker run
  • docker rmi

This list should eventually be comprehensive; anywhere you can refer to an image, you should be able to do so by digest. For the time being, the commands listed above should be sufficient for the use case listed in the overview.

docker images

If you pull an image from a v2 registry, the registry provides the image manifest's digest as a response header. If you pull by tag, you'll have both the tag and the digest. If you pull only by digest, you won't have the tag information. 1 question to resolve is how to display images, tags, digests, and v1 image IDs in the docker images command.

It is possible (likely?) that a variety of code exists to scrape the output of docker images. If we change the format, e.g. by adding a new column, or overloading an existing column to sometimes show a v1 image ID vs a digest, we will probably break whatever code is out there doing the scraping. We need to determine how critical it is to retain the existing column and data formats in the docker images command.

I have the following ideas regarding this output:

  • if an image was pulled by tag, and the registry it's pulled from includes the image's digest, display repository, tag, digest, and image id
  • if an image was pulled by digest, display repository, for tag, digest, and image id

docker images questions

  • should we add a new DIGEST column?
  • if yes, should we always display it, or only display it if you indicate that you want to see digests (e.g. via a filter)
    • there can be confusion if you pull an image by digest and then run docker images - you won't see the image you just pulled
  • should we display the digest instead of the v1 image ID (or some other value)?

Questions

What about v1 registry support?
The v1 registry won't support this feature.

If I create an image locally via docker tag or docker commit, can I refer to it by name@digest?
As proposed in docker/distribution#46, the registry is responsible for determining an image's digest and assigning it to the image. For an image that has not yet been pushed to a v2 registry, it may not be possible to refer to it by name@digest. This is unlikely to be a significant issue, as the use case for name@digest is consistent deployments using images pulled from registries. Or, if the community thinks this should be supported, we can revisit what component(s) are responsible for calculating digests.

Additional information

See #10740 for more backstory.

Todo:

  • Test cases
  • Update documentation
@GordonTheTurtle

Can you please sign your commits following these rules:

https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work

The easiest way to do this is to amend the last commit:

$ git clone -b "pull-by-digest" git@github.com:ncdc/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~3
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

This will update the existing PR, so you do not need to open a new one.

@ncdc
Contributor
ncdc commented Mar 2, 2015

Requires docker/distribution#211 in the v2 registry

Test cases are still being written, but I wanted to submit this now given the March 18th code freeze so the maintainers can begin to review it and provide feedback.

@icecrime @jfrazelle @crosbymichael @stevvooe @vbatts

@dmcgowan
Member
dmcgowan commented Mar 3, 2015

Can you list some example cli use cases. I see NAME[:TAG|@DIGEST] but don't see where @ symbol would be used. Since this PR is still in the design-review state, would be helpful to enumerate what visible changes are.

@ncdc
Contributor
ncdc commented Mar 3, 2015

Sure thing! Basically any place you potentially want to refer to an image using a tag, you can do it using a digest:

docker push localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
docker pull localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
docker run localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
docker create localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
docker rmi localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8

and in a Dockerfile:

FROM localhost:5000/andy/busybox@sha256:2aac5e7514fbc77125bd315abe9e7b0257db05fe498af01a58e239ebaccf82a8
@ncdc
Contributor
ncdc commented Mar 3, 2015

I also think we probably need to update the responses from the daemon to the client to include the digest (so things like go-dockerclient and other libraries can use it). This would be for push and pull initially, and later on possibly for build and tag as well (if those commands end up generating manifests at some point).

And we may want to display the digest to stdout as well.

@dmcgowan
Member
dmcgowan commented Mar 3, 2015

Can you define a few of these other changes as well...

What happens when deleting all tags?
...I observed the image is gone, but digest still remains in /var/lib/docker/repositories-(driver)

What happens when pulling an image which does not exist locally?
...I got a dangling image, which may be expected, however the UI is does not show the REPOSITORY
<none> <none> e45a5af57b00 8 weeks ago 910 B
These images would then also get cleaned up with docker rmi $(docker images -q --filter=dangling=true)

@ncdc
Contributor
ncdc commented Mar 3, 2015

What happens when deleting all tags?
...I observed the image is gone, but digest still remains in /var/lib/docker/repositories-

This is one I wanted input on. This is what's coded now, but I'm not sure what the appropriate desired behavior should be. I could use some help figuring out what makes the most sense here.

What happens when pull an image which does not exist locally?
...I got a dangling image, which may be expected, however the UI is does not show the REPOSITORY
e45a5af57b00 8 weeks ago 910 B
These images would then also get cleaned up with docker rmi $(docker images -q --filter=dangling=true)

I don't like this behavior, and am hoping we can come to a decision on what to do in this situation. We could

  • not display "dangling" images that were pulled by digest but don't have any corresponding tag pointing to them
  • display these images, perhaps like this
REPOSITORY TAG DIGEST IMAGE ID CREATED VIRUTAL SIZE
busybox latest sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69 4986bf8c1536 8 weeks ago 2.43 MB
busybox sha256:9bcaeba3d43193c3a2b61f5dc70e14d76e194f4766506b57610a10619919b192 4d2ed1de264f 8 weeks ago 2.43 MB

That makes the output quite wide, however, and we don't yet support shortened references to digests (TBD in docker/distribution#194).

  • consider adding a new flag to docker images or a new command to display digests

Do any of these sound good, or does anyone have other ideas?

@dmcgowan
Member
dmcgowan commented Mar 3, 2015

Currently the digests are only computed on the registry side, so then most the Digest fields would be blank unless all images were push or pulled with a v2 registry. For the second case I think I would prefer to just see the Repository and <none>. I also like the idea of having an optional argument to display digests.

My only worry with the first case is that it could lead to garbage building up in that file. I think removing the Digest link when the image is removed is the right thing to do. But then that relates to the second issue, what does it mean for an image to exist locally without a tag (why does rmi last tag delete an image and not put it in the same state as pulling without a tag?).

Will need consensus from @icecrime @jfrazelle @crosbymichael

@ncdc
Contributor
ncdc commented Mar 3, 2015

In order to be able to write integration tests for this, we'll need docker/distribution#211 merged, then we'll need to bump the commit in the Dockerfile for the distribution repo so we can spin up a v2 registry with support for pull by digest.

@ncdc
Contributor
ncdc commented Mar 3, 2015

I'm copying this comment from #10740 here as this is a better place for the discussion:

@stevvooe wrote in #10740 (comment) about having a parser that can return an "image object reference". Right now, parsers.ParseRepositoryTag takes a string, parses it, and returns a repository string and a tag string. In my prototype for this feature, I modified this method to return either the digest or the tag in the 2nd returned string. Doing it this way means that the remaining changes to support referring to images either by tag or by digest relatively minimal; however, it does muddy the waters a bit, since ParseRepositoryTag is now returning something that is either a digest or a tag. I've thought about a few possibilities for making this cleaner:

  1. Rename ParseRepositoryTag to ParseRepositoryReference, so it's clearer that it's not always a tag that comes back
  2. Do the rename from above, but also modify the signature to return (repository, tag, digest), where only 1 of tag and digest is ever set at a time
  3. Return a type, perhaps called ImageReference, that looks like this:
type ImageReference struct {
  Repository string // or possibly registry.RepositoryInfo instead of string
  Tag string
  Digest string
}

This ImageReference option would be a more invasive change, as anywhere ParseRepositoryTag is called will have to be modified to work with a struct instead of 2 strings.

@stevvooe indicated his desire to see ImageReference in #10740 (comment). Does anyone else want to weigh in?

@stevvooe stevvooe commented on an outdated diff Mar 3, 2015
registry/session_v2.go
@@ -12,6 +12,8 @@ import (
"github.com/docker/docker/utils"
)
+const DockerDigestHeader = "Docker-Digest"
@stevvooe
stevvooe Mar 3, 2015 Contributor

Would we prefer this as Docker-Content-Digest?

cc @ncdc

@stevvooe
Contributor
stevvooe commented Mar 3, 2015

@ncdc This is really up to the use case. In the digest package, where the string is mostly used as a unit, but is sometimes pulled apart, I've used the string + methods approach. If the primary use case is to use the exploded components, the struct definition is appropriate. I cannot tell from this PR which is better.

@shaded-enmity

+1 For the ImageReference, my original rationale for a class like this was that the logic that handles repositories and tags (+now digests) is largely duplicated throughout Docker codebase - just see how many times in the code appears repoName + ":" + tag or in how many places a default tag is appended to the string, which might be easily handled as ir.String() in one place.

As for the output format, I think that changing the default output of docker images is dangerous, people may already be parsing it and the digest addition makes it hardly readable (sure depends on actual width of your terminal).
What do you think about adding a switch, say, docker images --provenance or docker images --identifiers that'd produce non-truncated output by default:

Image ID Tag(s) Digest Repository
4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125 latest, newest sha256:9bcaeba3d43193c3a2b61f5dc70e14d76e194f4766506b57610a10619919b192 busybox

The output is: Image ID, All local tags for the given ID, Digest, RepoName

@ncdc
Contributor
ncdc commented Mar 3, 2015

I agree that changing the output of docker images could potentially break a lot of scraping scripts. I'm in favor of a new flag as well.

@ncdc
Contributor
ncdc commented Mar 3, 2015

I've done a bit of work attempting to replace ParseRepositoryTag with ParseImageReference. Given that 1.6 code freeze is 15 days away, I think it's too risky a change to try to include in this PR. There are places, such as graph/tags.go#Delete() where the image reference is actually a v1 image ID, and there is logic in Delete that keys off of that scenario to support delete-by-v1-image-ID. And that's just 1 example.

While it would be great to have, creating ImageReference as part of this PR increases complexity, QA scope, and risk. I would like to get the OK to defer that until after 1.6. Thoughts?

@stevvooe @icecrime @jfrazelle @crosbymichael @vbatts

@vbatts
Contributor
vbatts commented Mar 3, 2015

@ncdc while I think the ImageReference would be more clear and would help in abstracting the v1 and v2 references, it's probably best to hold off on that while the manifest discussion continues. There will likely be reusable structures that come from that work, and then we can later re-use them.

For now lets get this ready for the 1.6 window.

@dmcgowan
Member
dmcgowan commented Mar 3, 2015

Is push by digest intended to be part of this PR?

Along the same lines may also be tag by digest. Currently to push an image I pulled by digest, I must tag by image id then push using that tag.

@ncdc
Contributor
ncdc commented Mar 3, 2015

Push by digest is supposed to be part of this PR. If it's not working, I might have overlooked testing that specific feature. I don't think it's critical to our workflow (OpenShift), but I think it makes sense to include it.

Are you saying that docker tag foo/bar@<digest> <new name> isn't working? That wasn't something that I tested, so there's probably a code path that I missed.

@dmcgowan
Member
dmcgowan commented Mar 3, 2015

I don't think push by digest should be part of this PR and rather excluded until a later version which supports schema version 2 of the manifest. The current version of the manifest requires a tag to be set. Since the digest represents a manifest which contained a tag, that tag is what should be used for the push. However pull by digest currently does not use the manifest tag (perhaps it should) and it does not store the manifest. Since the manifest is not stored, pushing by digest will always recreate a new manifest and tars to push, causing a new digest to be created for the image id. @jlhawn is also working on a new implementation of storage of layers and manifest which would enable deterministic pushing of layers and manifests. It is probably best to wait until a new storage backend to support this.

The tag command should be supported if the digests are going to stick around for docker run

@ncdc
Contributor
ncdc commented Mar 3, 2015

I'm happy to omit push by digest for now.

FYI, pushing the same manifest content multiple times should result in the same digest, as the unsigned portion of the (recreated) manifest should remain the same if nothing else has changed.... right? That's what I'm seeing locally, at least.

The tag command should be supported if the digests are going to stick around for docker run

I'm not against adding support for tagging a digest to a new tag, but I'm not sure how docker run plays into this?

@dmcgowan
Member
dmcgowan commented Mar 3, 2015

We are not guaranteeing in this version of the manifest that the unsigned portion will not change. It shouldn't but there have been issues with using tarsum.

I just meant that if the digest is intended to be used for commands after pull, such as docker run, the image should be taggable.

@ncdc
Contributor
ncdc commented Mar 3, 2015

I just meant that if the digest is intended to be used for commands after pull, such as docker run, the image should be taggable.

We can certainly do that, but, at least for OpenShift, we plan on passing name@digest to docker run.

@ncdc
Contributor
ncdc commented Mar 3, 2015

@dmcgowan I'm able to docker tag <name>@<digest> <new name> - are you not?

@dmcgowan
Member
dmcgowan commented Mar 4, 2015

@ncdc just tested, yes I was able to, my initial test must have swapped : for @

@ncdc ncdc referenced this pull request in kubernetes/kubernetes Mar 4, 2015
Merged

Implementing ImageManager to take over image lifecycle. #4943

@ncdc
Contributor
ncdc commented Mar 4, 2015

I've added an initial set of integration-cli tests. PTAL.

Also, what other sorts of tests might we need? Perhaps something around the TagStore, now that I've added a 2nd top-level key to it (Digests)?

@ncdc
Contributor
ncdc commented Mar 4, 2015

I've got some code that I haven't committed that does this for docker images so images only pulled by digest don't show up as dangling:

REPOSITORY                         TAG                                                                       IMAGE ID            CREATED              VIRTUAL SIZE
docker                             pull-by-digest                                                            5ad5270479e6        11 seconds ago       1.557 GB
localhost:5000/andy/busybox        sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        8 weeks ago          2.43 MB

I'm not sure I'm crazy about it, since it shows the digest as a "tag", but it at least keeps it from being dangling. Thoughts?

@ncdc
Contributor
ncdc commented Mar 4, 2015

In my most recent commit, I'm now excluding images referenced by digest from the dangling images list, and if you pass -d or --digests to docker images, it will show the digest in the TAG column now.

@stevvooe
Contributor
stevvooe commented Mar 5, 2015

@ncdc Why wouldn't we show the digests in the image id column?

@dmcgowan
Member
dmcgowan commented Mar 5, 2015

@stevvooe these images still have an image ID which represents Docker's internal identifier. We shouldn't hide that image id since it is used by most commands including rmi.

@ncdc
Contributor
ncdc commented Mar 5, 2015

We could add another column for digest if you pass -d, perhaps?

@stevvooe
Contributor
stevvooe commented Mar 5, 2015

@ncdc That might be a good stopgap. We should come up with a plan to move people to the more secure digests over time.

@ncdc
Contributor
ncdc commented Mar 5, 2015

@stevvooe presumably when the daemon shifts to manifests as first class citizens?

@icecrime icecrime added this to the 1.6.0 milestone Mar 5, 2015
@stevvooe
Contributor
stevvooe commented Mar 5, 2015

@ncdc Yes, that would have to be the plan.

@vbatts
Contributor
vbatts commented Mar 5, 2015

I like that. Where would the best place be to document this? cli.md?

On Wed, Mar 4, 2015 at 9:44 PM, Stephen Day notifications@github.com
wrote:

@ncdc https://github.com/ncdc Yes, that would have to be the plan.


Reply to this email directly or view it on GitHub
#11109 (comment).

@ncdc
Contributor
ncdc commented Mar 5, 2015

Ok, here's what I have locally.

With an image pulled by digest, with no tags referring to it:

REPOSITORY                    TAG                 DIGEST                                                                    IMAGE ID            CREATED             VIRTUAL SIZE
localhost:5000/andy/busybox                       sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        9 weeks ago         2.43 MB

With 1 tag referring to it:

REPOSITORY                    TAG                 DIGEST                                                                    IMAGE ID            CREATED             VIRTUAL SIZE
localhost:5000/andy/busybox   latest              sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        9 weeks ago         2.43 MB

And multiple repos/tags referring to the same image:

REPOSITORY                         TAG                                DIGEST                                                                    IMAGE ID            CREATED              VIRTUAL SIZE
busybox                            latest                             sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        9 weeks ago          2.43 MB
localhost:5000/andy/busybox        latest                             sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        9 weeks ago          2.43 MB

Note: the current relationship is named repository -> manifest digest -> v1 image ID. Therefore, if multiple tags all refer to the same v1 image ID, they are all considered to share the same manifest digest.

@ncdc
Contributor
ncdc commented Mar 5, 2015

Also note: this latest commit adds 2 new fields to the items returned by list images: Digest and Repo. Digest will always be populated if it's available for the given image. Repo will only be populated if the image has no tag (i.e. just a digest). I'd like to find a backwards-compatible way to do this, so we don't have to modify downstream clients such as go-dockerclient, but that may not be possible if we want to be able to show tag and digest for those images that have both.

@vbatts
Contributor
vbatts commented Mar 5, 2015

should we use the <none> marker when there is no tag? for an awk '{ print $2 }' would grab the digest column ...

@ncdc
Contributor
ncdc commented Mar 5, 2015

I can do that

@stevvooe
Contributor
stevvooe commented Mar 6, 2015

@ncdc 👍

@ncdc
Contributor
ncdc commented Mar 6, 2015

From my comment above:

Note: the current relationship is named repository -> manifest digest -> v1 image ID. Therefore, if multiple tags all refer to the same v1 image ID, they are all considered to share the same manifest digest.

This is actually a problem. Let's take my example from above:

REPOSITORY                         TAG                                DIGEST                                                                    IMAGE ID            CREATED              VIRTUAL SIZE
busybox                            latest                             sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        9 weeks ago          2.43 MB
localhost:5000/andy/busybox        latest                             sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69   4986bf8c1536        9 weeks ago          2.43 MB

To do this, I ran the following sequence of commands:

docker pull busybox:latest
docker tag busybox:latest localhost:5000/andy/busybox:latest
docker push localhost:5000/andy/busybox:latest

At this point, busybox:latest and localhost:5000/andy/busybox:latest both point to the same v1 image ID. When I pushed localhost:5000/andy/busybox:latest to the registry, it got the digest sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69. In the docker images -d output above, this digest is shown for both busybox:latest (which is incorrect) and localhost:5000/andy/busybox:latest (which is valid).

If I try to run the following command:

docker run --rm -it busybox@sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69

it will fail, because the TagStore only has a mapping for localhost:5000/andy/busybox@sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69 right now, which makes sense, since that digest came from my previous push of that image to my v2 registry.

If I run the following command, it will succeed:

docker run --rm -it localhost:5000/andy/busybox@sha256:6b019df8c73bb42e606225ef935760b9c428521eba4ad2519ef3ff4cdb3dbd69

I potentially could modify Digests in the TagStore to include tag information, which would help display the output correctly, but I'm not sure if that's the right direction. @stevvooe what do you think?

@ncdc
Contributor
ncdc commented Mar 6, 2015

I just pushed a new commit that fixes the issue in the comment above. Now the digest is only shown with the appropriate name+tag. If you have a digest that isn't associated with a tag, it will show <none> for the tag. Even though with schema version 1, the manifest includes the tag, so we could show it, I think it's better not to.

@stevvooe
Contributor
stevvooe commented Mar 6, 2015

@ncdc Doh! I had a whole response for you but forgot to submit.

Part of the issue here is that with v2 manifest, schema version 1, the name and the tag are part of the manifest. If we are doing this correctly, "busybox" and "localhost:5000/andy/busybox" should really have different digests, since they have different names and different tags.

I think you're fix covers this.

@ncdc
Contributor
ncdc commented Mar 6, 2015

FYI, this is what the Digests entry in the TagStore looks like as json:

    "Digests": {
        "localhost:5000/andy/busybox": {
            "sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf": {
                "Tag": "another",
                "V1ImageID": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125"
            }
        }
    },
@vbatts
Contributor
vbatts commented Mar 6, 2015

Having the Tag down in there is less than fun. Ideally we'll be able to get that massaged out as we migrate to v2, but for now it'll be there for version interoperability.

This is shaping up to an LGTM from me. Perhaps a blurb in the cli.md docs?

@ncdc
Contributor
ncdc commented Mar 6, 2015

@vbatts et al, yeah, I want to know which of the various pieces of documentation need updating...

@vbatts
Contributor
vbatts commented Mar 6, 2015

https://github.com/docker/docker/blob/master/docs/sources/reference/commandline/cli.md

first off, images and rmi

On Fri, Mar 6, 2015 at 4:16 PM, Andy Goldstein notifications@github.com
wrote:

@vbatts https://github.com/vbatts et al, yeah, I want to know which of
the various pieces of documentation need updating...


Reply to this email directly or view it on GitHub
#11109 (comment).

@stevvooe stevvooe was assigned by dmp42 Mar 7, 2015
@vbatts
Contributor
vbatts commented Mar 9, 2015

docs are looking good. Should we allude to more info on the what the digest is? like to v2 docs?

@ncdc
Contributor
ncdc commented Mar 9, 2015

I was considering adding a section to the top of cli.md describing image references (v1 id, tag, digest). Maybe we could do that and also add a ref to the docs in docker/distribution?

@stevvooe
Contributor
stevvooe commented Mar 9, 2015

@ncdc Was considering add a small specification for the digest format. It's relatively simple but I'd like to make it consistent.

@ncdc
Contributor
ncdc commented Mar 9, 2015

@stevvooe is that addition of the spec something you'll tackle?

@icecrime
Member
icecrime commented Mar 9, 2015

I'm gonna be honest, I fail to review all the implications of that change. That being said, I attempted to use it to wrap my head around it, and I get very unexpected behavior.

  • Pulled busybox, pushed to localhost:5000/icecrime/busybox
  • docker rmi all images
  • Find busybox sha256 in registry storage
$ docker pull localhost:5000/icecrime/busybox@sha256:0ecb2ad6099635efbc9715bcc26c9bbe33df1227a54da0b70cee9bbccd099aae
sha256:0ecb2ad6099635efbc9715bcc26c9bbe33df1227a54da0b70cee9bbccd099aae: Pulling from localhost:5000/icecrime/busybox

511136ea3c5a: Pull complete
df7546f9f060: Pull complete
ea13149945cb: Pull complete
4986bf8c1536: Pull complete
Digest: sha256:0ecb2ad6099635efbc9715bcc26c9bbe33df1227a54da0b70cee9bbccd099aae
Status: Downloaded newer image for localhost:5000/icecrime/busybox@sha256:0ecb2ad6099635efbc9715bcc26c9bbe33df1227a54da0b70cee9bbccd099aae

root@9238bbccfe45:/go/src/github.com/docker/docker# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

I don't understand why nothing shows up after a presumably successful pull.

@ncdc
Contributor
ncdc commented Mar 9, 2015

@icecrime if you pull by digest, the manifest has no tag associated with it (well, it does now, but when we get the new v2 schemaVersion, it won't). If you want to see images that you've only pulled by digest, you need to do docker images -d

@vbatts
Contributor
vbatts commented Mar 10, 2015

@icecrime that is the expected behavior.

Besides meeting immediate production use cases (not at all limited to openshift), this is semantically on the same path toward the v2 format and having everything be content addressable. There is a small amount to wrap your head around, but makes sense and I feel can be conveyed in the docs.

Further, this will largely be a transparent feature for the 80% of users, and huge win to the rest.

@icecrime @dmcgowan @stevvooe can we review this feature? Winter is coming (the freeze day), and this is quite needed.

@icecrime
Member

I see, sorry for the missing -d, should have read the doc first.

  • How am I supposed to know about a given image digest in the first place? I tried pulling ubuntu and listing with docker image -d, I don't have its digest.
  • I find the requirement for -d misleading, and docker images -a doesn't even show those: we have to figure out something better. I can't help but think that digest is somehow comparable to tag (which today is just a "named digest" as I understand it), and that I wouldn't be surprised to see the x first bytes of a sha256 hash as my image TAG when I pull by digest, but maybe that's even worse...
@tianon
Member
tianon commented Mar 10, 2015

Does -d filter the list to only images that have a digest, or does it change the list too (for example to include the DIGEST column)? It seems confusing to me to add new arguments that seem like filters when we have a generic filters interface that @vbatts constructed.

@ncdc
Contributor
ncdc commented Mar 10, 2015

@icecrime

How am I supposed to know about a given image digest in the first place? I tried pulling ubuntu and listing with docker image -d, I don't have its digest

Right now, digests only exist when you push to or pull from a v2 registry that has commit docker/distribution@b4cc5e3 or later.

I find the requirement for -d misleading, and docker images -a doesn't even show those: we have to figure out something better. I can't help but think that digest is somehow comparable to tag (which today is just a "named digest" as I understand it), and that I wouldn't be surprised to see the x first bytes of a sha256 hash as my image TAG when I pull by digest, but maybe that's even worse...

I was hoping to be able to do the following with images, tags, and digests:

  • if an image was pulled by tag, and the registry it's pulled from includes the image's digest, display repository, tag, digest, and image id
  • if an image was pulled by digest, display repository, <none> for tag, digest, and image id

If that logic is incorrect or undesirable, I would love to correct it to something that works for everyone. Please let me know what makes the most sense.

Also, we currently have to refer to images by digest using the full value - truncation is not currently supported (see docker/distribution#194 for more details). Given the long length of each digest, and my desire to show tag and digest if both are available, it seemed to make the most sense to provide some way to "turn on" showing digests and keep them hidden by default. I'm happy to change this to whatever works best for everyone.

@tianon

Does -d filter the list to only images that have a digest, or does it change the list too (for example to include the DIGEST column)?

It shows all images you'd see with a normal docker images call, as well as those that were only pulled by digest and don't have a known tag. It also adds a DIGEST column.

It seems confusing to me to add new arguments that seem like filters when we have a generic filters interface that @vbatts constructed.

I agree and I will remove -d and convert it to a filter.

@ncdc
Contributor
ncdc commented Mar 10, 2015

Before I convert -d to a filter, I guess we should agree upon the desired UX. What are your thoughts on how/when to display digests, what to do about columns, etc?

@termie
Contributor
termie commented Mar 10, 2015

-subscribe-

@icecrime
Member

@ncdc I think your suggestion makes sense: display all available information (so pulling from non v2 won't include digest, and pulling by digest won't include a tag). I'd rather show this unconditionally, however 1) not being able to truncate digest is a blocker 2) it makes it difficult to grasp the difference between ImageID and Digest

@ncdc
Contributor
ncdc commented Mar 10, 2015

not being able to truncate digest is a blocker

@stevvooe @dmcgowan any thoughts here?

@dmcgowan
Member

I like the idea of always showing the digest, it is a good start in transitioning away from the random image identifiers. The blocker @icecrime mentioned will need to be addressed in future versions by updating the tag/graph store to store the actual manifests. Then the digests will be reproducible by content on disk. I have concerns related to parsing if the digest is always showing before the image ID. If scripts are parsing the output, it is likely reading the 3rd column to get the image identifier. Even though it makes more sense to show the digest inbetween the tag and image id, I wonder if it is better to put it after the image id, if it will always be shown. Of course using --filter and -q is the preferred way to get the image id via a script, but its still a use case that will effect users.

@dmcgowan
Member

not being able to truncate digest is a blocker

We can certainly show a truncated version, we just need to detect ambiguous identifiers. When operations on done on digests that are referenced locally, it shouldn't be an issue to find which digest is referenced. If multiple digests match, it is an error. If this is really the only blocker we can focus on adding short digest utilities to make this easier.

@ncdc
Contributor
ncdc commented Mar 11, 2015

When operations on done on digests that are referenced locally, it shouldn't be an issue to find which digest is referenced

In the current form, in my PR, it will involve iterating through every key (repo) in TagStore.Digests to get to the map of digests->digest mappings, then iterating through every key in that map to look at each digest.

@dmcgowan
Member

I was thinking this only needs to be done in the context of a single image name. Then for operations on repo/myapp@4713a7b2 would only need to look at all the digests for repo/myapp and see if one (and only one) of them start with 4713a7b2. Obviously a pull would currently fail using this syntax because it wouldn't know how to deference, however in the future if we have a trusted list of tagged images it could reference from there.

@ncdc
Contributor
ncdc commented Mar 11, 2015

Sure, makes sense 😄

@ncdc
Contributor
ncdc commented Mar 11, 2015

Summary of proposed changes to docker images:

  • add a new DIGEST column and always show it (i.e. no need for a filter or --digests option)
  • place the DIGEST column after IMAGE ID
  • if an image doesn't have a digest, show <none>
  • when showing the digest, truncate it similar to how v1 image ids are truncated

Additionally, add support for referring to digests using the truncated format. This will currently only work for digests that exist locally in the TagStore metadata (i.e. for images have been pushed to or pulled from a v2 registry). If there are multiple matches when a user attempts to refer to a digest using the shortened form, return an error indicating the user needs to be more specific to uniquely identify the desired digest.

@crosbymichael
Member

LGTM for change of specifying images to download by digest namespace/repository@digest. Just remember that this should be exactly what you want as you will not be able to change this format as it will be used in Dockerfiles and various references. Make sure the format is what you want before merging.

Review and stability for this is the responsibility of the distribution maintainers, feel free to merge when it's complete. Moving the status to code-review. Thanks for the PR @ncdc .

@ncdc
Contributor
ncdc commented Mar 11, 2015

Updated proposed changes to docker images based on our meeting today:

  • do not show digests by default
  • show all images with tags or pulled by digest by default
  • when showing an image that was pulled by digest (has no tag), show <none> for the tag
  • if you want to see image digests, run docker images --digests, and in this mode, a new DIGESTS column will be added
  • for now, show the full digest (do not truncate)
@vbatts
Contributor
vbatts commented Mar 12, 2015

all the changes detailed in #11109 (comment) look good. @dmcgowan @stevvooe WDYT?

@ncdc
Contributor
ncdc commented Mar 12, 2015

@vbatts we actually changed our minds during our meeting today - see #11109 (comment)

@dmcgowan
Member

@vbatts had you not totally ditched us earlier, you would know we agree with the comment Andy just pointed to.

As for shortened digests, we agreed digest will not be exposed by default in this version. We are going to work on a short digest but it would likely be added on in a separate commit and is only for sugar, not a blocker for using or removing by digests.

@stevvooe
Contributor

@ncdc List looks good to me.

@jlhawn jlhawn referenced this pull request Mar 12, 2015
Closed

Add support for referring to images by digest #11341

2 of 2 tasks complete
@vbatts
Contributor
vbatts commented Mar 12, 2015

/me is sick-burned by @dmcgowan

@vbatts
Contributor
vbatts commented Mar 12, 2015

ohman i see now that i also commented on an un-refreshed tab.

/me deserves the sick-burn

@ncdc
Contributor
ncdc commented Mar 12, 2015

With @jlhawn's commit, the behavior of docker push and docker pull changed so that if you push/pull by tag, the digest is not stored by the daemon. The daemon only stores the digest if you pull by digest. My concern here is that unless you pay attention to the docker pull output, you have no way of getting the digest after the fact for an image you've pulled.

This alters the behavior I originally had in my branch, so I took a stab at restoring it... But we now have this (probably undesirable) behavior:

$ docker pull foo:latest
$ docker images --digests
# you'll see 2 entries, 1 for repo=foo, tag=latest, digest=<none>, 1 for repo=foo, tag=<none>, digest=$digest
$ docker rmi foo:latest

With the rmi call, this untags foo:latest but doesn't delete the image because TagStore.Repositories still has a mapping from foo -> $digest -> $imageID.

When we had TagStore.Digests and TagStore.Repositories, the logic I had for rmi was to delete the digest entries if you removed the last tag for a repo.

@dmcgowan @jlhawn @vbatts @icecrime @stevvooe @dmp42 what do you all think we should do?

@ncdc
Contributor
ncdc commented Mar 12, 2015

Another issue that I just found that shows up with Josh's branch and the tip of mine:

$ docker pull foo:latest
$ docker pull foo@sha256:...
$ docker run -d foo:latest true
$ docker rmi foo:latest

The docker rmi call succeeds, untagging foo:latest when it should print out an error saying a container is using the image, and require the use to pass -f to force deletion.

The first pull of foo:latest adds an entry to TagStore.Repositories:

"foo": {
  "latest": "<imageid>"
}

The second pull of foo@sha256:... adds a second entry to the map, so it looks like this:

"foo": {
  "latest": "<imageid>"
  "sha256:...": "<imageid>"
}

The problem with this is now that when you docker rmi foo:latest, this condition (https://github.com/docker/docker/blob/master/daemon/image_delete.go#L91-L95) fails because repos has 2 entries for <imageid>: latest and sha256:.... As a result, since canDeleteImage() is not called, the rmi is allowed to proceed and untags foo:latest without any warnings.

@vbatts
Contributor
vbatts commented Mar 12, 2015

I feel like we are now bridging a transitional period of behavior. What i'll call existing/legacy and future/v2.

Perhaps this should require a new daemon flag that we presently will default to existing/legacy, and allow for future/v2. Being that existing prefers image-id/Tags, and future/v2 prefers digests.

That seems like the modal behavior differences that we're needing to define.

@ncdc
Contributor
ncdc commented Mar 12, 2015

I think the question boils down to this: when we're in a situation where image digests are stored by the daemon in the TagStore, how should we handle requests to delete by tag?

The pre-digest behavior is that when you remove a tag, and it's the last reference to an image id, that image id (and dangling parents) is removed.

With digests, we now have 2 different ways to refer to an image id: by tag and by digest. If you remove a tag, and it's the last tag reference to an image id, but there are still digest references to that image id, what should we do?

Related question - in my branch (prior to pulling in Josh's work), if you remove the last tag for a repo, should it also remove all the digest info for that repo?

Sounds like we need another call...

@jlhawn
Contributor
jlhawn commented Mar 12, 2015

With @jlhawn's commit, the behavior of docker push and docker pull changed so that if you push/pull by tag, the digest is not stored by the daemon. The daemon only stores the digest if you pull by digest. My concern here is that unless you pay attention to the docker pull output, you have no way of getting the digest after the fact for an image you've pulled.

This was what I intended. If you pull by tag then there's only a tag reference. If you pull by digest then there's only a digest reference. The typical user does not expect that pulling by a tag would actually result in 2 references to the same image. I wanted to make it simple: you only get the reference you asked for - tag or digest.

My concern here is that unless you pay attention to the docker pull output, you have no way of getting the digest after the fact for an image you've pulled.

For now, users (or automated scripts) who want to get the digest reference for a tag they pushed/pulled should pay attention to the Digest: ... line in Pull or Push output.

When we had TagStore.Digests and TagStore.Repositories, the logic I had for rmi was to delete the digest entries if you removed the last tag for a repo.

This also seems undesirable to me - was there an option to remove only the tag and keep the digest?

Another issue that I just found that shows up with Josh's branch and the tip of mine:
...
The docker rmi call succeeds, untagging foo:latest when it should print out an error saying a container is using the image, and require the use to pass -f to force deletion.
...
The problem with this is now that when you docker rmi foo:latest, this condition (https://github.com/docker/docker/blob/master/daemon/image_delete.go#L91-L95) fails because repos has 2 entries for : latest and sha256:.... As a result, since canDeleteImage() is not called, the rmi is allowed to proceed and untags foo:latest without any warnings.

While this seems unusual it is no different than the behavior in the docker engine today: Containers do not run named images they run image IDs. Here's an example (I'm running Docker 1.5):

$ docker pull busybox:latest
...
$ docker tag busybox:latest busybox:A
$ docker tag busybox:latest busybox:B
$ docker images busybox
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
busybox             A                   4986bf8c1536        10 weeks ago        2.433 MB
busybox             B                   4986bf8c1536        10 weeks ago        2.433 MB
busybox             latest              4986bf8c1536        10 weeks ago        2.433 MB

I've got 3 references to the same Image ID now: busybox:latest, busybox:A, and busybox:B. If I start a container using each of those references:

$ docker run -d busybox:latest
    ...
$ docker run -d busybox:A
    ...
$ docker run -d busybox:B
    ...
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
b80614ebc706        busybox:A           "/bin/sh"           2 seconds ago       Exited (0) 2 seconds ago                        insane_bartik       
bab45219b86d        busybox:A           "/bin/sh"           4 seconds ago       Exited (0) 3 seconds ago                        goofy_stallman      
313a5da8117f        busybox:A           "/bin/sh"           15 seconds ago      Exited (0) 15 seconds ago                       reverent_franklin

Oh, look! docker ps says all 3 containers are running busybox:A! Why? Well, they are all actually running Image ID 4986bf8c1536... but the daemon will find the first tag that points to that ID (if one exists) to use for display purposes.

What if I remove the busybox:A tag?

$ docker rmi busybox:A
Untagged: busybox:A
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
b80614ebc706        busybox:B           "/bin/sh"           3 minutes ago       Exited (0) 3 minutes ago                       insane_bartik       
bab45219b86d        busybox:B           "/bin/sh"           3 minutes ago       Exited (0) 3 minutes ago                       goofy_stallman      
313a5da8117f        busybox:B           "/bin/sh"           3 minutes ago       Exited (0) 3 minutes ago                       reverent_franklin

Now the containers are displayed as running busybox:B! Let's repeat.

$ docker rmi busybox:B
Untagged: busybox:B
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
b80614ebc706        busybox:latest      "/bin/sh"           4 minutes ago       Exited (0) 4 minutes ago                       insane_bartik       
bab45219b86d        busybox:latest      "/bin/sh"           4 minutes ago       Exited (0) 4 minutes ago                       goofy_stallman      
313a5da8117f        busybox:latest      "/bin/sh"           4 minutes ago       Exited (0) 4 minutes ago                       reverent_franklin

Now they are running busybox:latest. Removing this last tag:

$ docker rmi busybox:latest
Error response from daemon: Conflict, cannot delete 4986bf8c1536 because the container b80614ebc706 is using it, use -f to force
FATA[0000] Error: failed to remove one or more images

Only now did the daemon find that no more references to the image ID exist and it tried to delete the Image ID, but can't because the container is using it.

So again: containers don't use named repo/tag/digest images - they use Image IDs.

What would we like? We'd like repository@digest references to be the core concept with tags resolving to a digest, but for now the Docker Engine is way too entrenched in using Image IDs everywhere. So until we have a Docker Engine 2.0 which switches to this image model I think it's best to treat tags and digests similarly as simple references to an Image ID. The distinction between tag and digest refs is that you can only get a digest reference in the engine by pulling a digest reference explicitly.

@ncdc
Contributor
ncdc commented Mar 12, 2015

Oh, look! docker ps says all 3 containers are running busybox:A

That's actually a bug, fixed by #10601

@jlhawn
Contributor
jlhawn commented Mar 12, 2015

That's actually a bug, fixed by #10601

Well, that's neat 👍

@ncdc
Contributor
ncdc commented Mar 12, 2015

I just ran your A/B/C test, and with 10061 merged in, you can actually rmi A when you shouldn't be able to :-(

@jlhawn
Contributor
jlhawn commented Mar 12, 2015

I just ran your A/B/C test, and with 10061 merged in, you can actually rmi A when you shouldn't be able to :-(

Yeah, I read the code and said to myself "wait a minute...".

@ncdc
Contributor
ncdc commented Mar 12, 2015

Ok, after talking to Josh, I think his approach makes the most sense - if you push/pull by tag, we don't store digest info (but it's displayed to stdout so you can make a note of it). If you push/pull by digest, we store digest info but not tag info. And the rmi issue mentioned above is not new behavior as a result of this PR.

@vbatts
Contributor
vbatts commented Mar 12, 2015

alrighty, then what is left then?

  • polish/squash commits
  • see why windows jenkins failed
  • clarify the agreed upon behavior in the docs
@jlhawn
Contributor
jlhawn commented Mar 12, 2015

@vbatts I took the liberty of adding fancy checkboxes to your comment ^ 😃

@jlhawn
Contributor
jlhawn commented Mar 12, 2015

LGTM

@stevvooe
Contributor

LGTM!

@dmcgowan
Member

Found a bug when attempting to rmi an image by the image ID when only a digests exists.

$ docker images localhost:5001/dmcgowan/busybuild
REPOSITORY                          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
localhost:5001/dmcgowan/busybuild   <none>              0aac67236a1e        6 weeks ago         2.43 MB
$ docker rmi 0aac67236a1e
Error response from daemon: No such repository: localhost:5001/dmcgowan/busybuild:sha256
FATA[0000] Error: failed to remove one or more images 
@ncdc
Contributor
ncdc commented Mar 12, 2015

I can look at it around 8pm my time. Or feel free to dive in, and I guess
we need a new test case?
On Thu, Mar 12, 2015 at 6:48 PM Derek McGowan notifications@github.com
wrote:

Found a bug when attempting to rmi an image by the image ID when only a
digests exists.

$ docker images localhost:5001/dmcgowan/busybuild
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
localhost:5001/dmcgowan/busybuild 0aac67236a1e 6 weeks ago 2.43 MB
$ docker rmi 0aac67236a1e
Error response from daemon: No such repository: localhost:5001/dmcgowan/busybuild:sha256
FATA[0000] Error: failed to remove one or more images


Reply to this email directly or view it on GitHub
#11109 (comment).

@dmcgowan
Member

@jlhawn Looks like the ByID function would need to be updated to return "repo@digest", currently it will append all digests as "repo:digest"

https://github.com/ncdc/docker/blob/pull-by-digest/graph/tags.go#L144

@dmcgowan
Member

This fixed it for me

diff --git a/graph/tags.go b/graph/tags.go
index 70e25e2..5d26b8c 100644
--- a/graph/tags.go
+++ b/graph/tags.go
@@ -147,7 +147,7 @@ func (store *TagStore) ByID() map[string][]string {
        byID := make(map[string][]string)
        for repoName, repository := range store.Repositories {
                for tag, id := range repository {
-                       name := repoName + ":" + tag
+                       name := utils.ImageReference(repoName, tag)
                        if _, exists := byID[id]; !exists {
                                byID[id] = []string{name}
                        } else {
@stevvooe stevvooe referenced this pull request in docker/distribution Mar 12, 2015
Closed

doc/spec: namespaces and discovery #179

@jlhawn
Contributor
jlhawn commented Mar 13, 2015

LGTM

@dmcgowan
Member

LGTM

@vbatts @stevvooe then we can move to docs-review

@stevvooe
Contributor

LGTM!

@icecrime
Member

Moving to 3-docs-review!

@ncdc
Contributor
ncdc commented Mar 13, 2015

@SvenDowideit @moxiegirl @fredlf this is in docs-review - please take a look. Thanks!

@jlhawn
Contributor
jlhawn commented Mar 13, 2015

WE WILL
WE WILL
MERGE YOU!

29076-brian-wilson-we-will-rock-you-98ku

@moxiegirl moxiegirl and 1 other commented on an outdated diff Mar 13, 2015
docs/sources/reference/api/docker_remote_api_v1.18.md
+ "Created": 1420064636,
+ "Id": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
+ "ParentId": "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
+ "RepoDigests": [
+ "localhost:5000/test/busybox@sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf"
+ ],
+ "RepoTags": [
+ "localhost:5000/test/busybox:latest",
+ "busybox:latest",
+ ],
+ "Size": 0,
+ "VirtualSize": 2429728
+ }
+ ]
+
+In the example above, this image ID is associated with 1 digest for the
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

The response shows a single image Id associated with two repositories (RepoTags): localhost:5000/test/busybox: and busybox A caller can use either of the RepoTags values localhost:5000/test/busybox:latest or busybox:latest to reference the image.

You can also use RepoDigests values to reference an image. In this response, the array has only one reference and that is to the localhost:5000/test/busybox repository; the busybox repository has no digest. You can reference this digest using the value: localhost:5000/test/busybox@sha256:cbbf2f9a99b47fc460d...

See the docker run and docker build commands for examples digest and tag references on the command line.

@ncdc
ncdc Mar 13, 2015 Contributor

@moxiegirl based on the example output, the image can by referenced by

  • localhost:5000/test/busybox:latest
  • busybox:latest
  • localhost:5000/test/busybox@sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf

It cannot be referenced by busybox@sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf

The point I am trying to make is that even though you pulled an image by digest (localhost:5000/test/busybox@sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf) and it happens to have the same v1 image id as busybox:latest, you can't just take the digest portion from 1 repo and move it to another and have it work.

@ncdc
ncdc Mar 13, 2015 Contributor

@moxiegirl how is this?

The response shows a single image Id associated with the repositories localhost:5000/test/busybox and busybox. A caller can use the localhost:5000/test/busybox:latest or busybox:latest tag in the RepoTags array to reference the image. A caller can also use the localhost:5000/test/busybox@sha256... digest reference in the RepoDigests array to reference the image.

@moxiegirl
moxiegirl Mar 13, 2015 Contributor

@ncdc @jlhawn came by to straighten me out. I think my rewrite is best for RUX (reader UX) Trying to explain all the components in (RepoTags, RepoDigests, repository, tag, digest) overloads the sentences to much. You want to focus the reader on one structure (RepoTags) and then the next (RepoDigest). Y'all can parse because you have been thinking about it way more than the average reader ever will.

Just thought of this, I think part of the confusion is busybox appears in both RepoTags are they always the same name --- probably huh-- or you can do this:

"RepoTags": [
   "localhost:5000/test/busybox:latest",
   "playdate:latest",
 ],
``
@ncdc
ncdc Mar 13, 2015 Contributor

@moxiegirl I'll do whatever you think makes the most sense. Do you want me to use playdate instead of busybox?

@ncdc
ncdc Mar 13, 2015 Contributor

I've replaced it with playdate locally. I think it's clearer.

@moxiegirl moxiegirl and 1 other commented on an outdated diff Mar 13, 2015
docs/sources/reference/api/docker_remote_api_v1.18.md
@@ -1042,6 +1042,38 @@ Status Codes:
}
]
+** Example request, with digest information**:
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

Remove the space between ** and the letter E as this is not resolving in display.

@ncdc
ncdc Mar 13, 2015 Contributor

Will be fixed in next push

@moxiegirl moxiegirl and 1 other commented on an outdated diff Mar 13, 2015
docs/sources/reference/builder.md
@@ -204,8 +208,8 @@ to start by **pulling an image** from the [*Public Repositories*](
multiple images. Simply make a note of the last image ID output by the commit
before each new `FROM` command.
-If no `tag` is given to the `FROM` instruction, `latest` is assumed. If the
-used tag does not exist, an error will be returned.
+If no `tag` or `digest` is given to the `FROM` instruction, the tag `latest` is
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

The tag or digest values are optional. If you omit either of them, the builder assumes the value latest by default. The builder returns an error if it cannot match the tag value.

@ncdc
ncdc Mar 13, 2015 Contributor

Will be fixed in next push

@moxiegirl moxiegirl and 1 other commented on an outdated diff Mar 13, 2015
docs/sources/reference/api/docker_remote_api_v1.18.md
+**Example response, with digest information**:
+
+ HTTP/1.1 200 OK
+ Content-Type: application/json
+
+ [
+ {
+ "Created": 1420064636,
+ "Id": "4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125",
+ "ParentId": "ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2",
+ "RepoDigests": [
+ "localhost:5000/test/busybox@sha256:cbbf2f9a99b47fc460d422812b6a5adff7dfee951d8fa2e4a98caa0382cfbdbf"
+ ],
+ "RepoTags": [
+ "localhost:5000/test/busybox:latest",
+ "busybox:latest",
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

remember to delete the bad comma

@ncdc
ncdc Mar 13, 2015 Contributor

Will be fixed in next push

@moxiegirl moxiegirl commented on the diff Mar 13, 2015
docs/sources/reference/commandline/cli.md
@@ -1136,6 +1138,20 @@ uses up the `VIRTUAL SIZE` listed only once.
tryout latest 2629d1fa0b81b222fca63371ca16cbf6a0772d07759ff80e8d1369b926940074 23 hours ago 131.5 MB
<none> <none> 5ed6274db6ceb2397844896966ea239290555e74ef307030ebb01ff91b1914df 24 hours ago 1.089 GB
+#### Listing image digests
+
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

Images that use the v2 or later format, have a content-addressable identifiers called a digest. As long as the input used to generate the image is unchanged, the digest value is predictable. To list image digest values, use the --digests flag:

When pushing or pulling to a 2.0 registry, the pushor pull command output concludes with the image digest.
You can pull using a digest value. You can also reference by digest in create,run, and rmi commands.

@moxiegirl moxiegirl and 1 other commented on an outdated diff Mar 13, 2015
docs/sources/reference/commandline/cli.md
@@ -1608,8 +1628,8 @@ deleted.
#### Removing tagged images
-Images can be removed either by their short or long IDs, or their image
-names. If an image has more than one name, each of them needs to be
+An image can be removed by its short or long ID, its tag, or its digest. If an
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

You can remove an image using its short or long ID, its tag, or its digest. If an image has one or more tag or digest reference, you must remove all of them before you can remove the image itself.

@ncdc
ncdc Mar 13, 2015 Contributor

@moxiegirl when you remove the last reference (tag or digest) to an image, the image data is removed. There isn't an additional rmi operation that you must perform after you've removed all of the references.

@moxiegirl moxiegirl and 1 other commented on an outdated diff Mar 13, 2015
docs/sources/reference/run.md
@@ -134,6 +134,11 @@ While not strictly a means of identifying a container, you can specify a version
image you'd like to run the container with by adding `image[:tag]` to the command. For
example, `docker run ubuntu:14.04`.
+### Image[@digest]
+
+Images using the v2 or later image format may be referred to using their
@moxiegirl
moxiegirl Mar 13, 2015 Contributor

Images that use the v2 or later format, have a content-addressable identifiers called a digest. As long as the input used to generate the image is unchanged, the digest value is predictable and referenceable.

@shaded-enmity
shaded-enmity Mar 13, 2015

@moxiegirl The second sentence bears a connotation that if I update the image content the former digest will no longer be accessible. Also I think that the digest value is always predictable after all (so that different registries arrive at the same digest for the same input regardless of any consensus protocol being involved).

Essentially I'd change the tone from negative, describing what digests are not, to positive, in the vein of:
" Digests are identifiers computed from both manifest metadata and image layer content, hence they provide a unique absolute identifier for the given input content & metadata."

@ncdc
Contributor
ncdc commented Mar 13, 2015

@moxiegirl comments addressed (with 1 minor addition about referring to a digest for FROM in a Dockerfile).

@moxiegirl
Contributor

LGTM

@dmcgowan
Member

@ncdc did some of the changes go missing? I am seeing that bug about rmi popping up again, the change to ByID in tags.go is not in the latest change.

@ncdc
Contributor
ncdc commented Mar 13, 2015

Yeah, my bad. I'll correct and re-push soon.

@ncdc
Contributor
ncdc commented Mar 13, 2015

Apologies for the mixup. @dmcgowan please re-review the code changes. @moxiegirl would you mind double-checking the docs again? Thanks!

@dmcgowan
Member

update LGTM

@moxiegirl
Contributor

update LGTM

@vbatts
Contributor
vbatts commented Mar 15, 2015

LGTM (despite the failure on windows)

@SvenDowideit
Collaborator

In the PR summary, you document what this feature is useful for - but this information is not reflected in the documentation changes you've made.

Can you perhaps add the use case example and explanation to the docs too?

@ncdc
Contributor
ncdc commented Mar 16, 2015

I can do that. Which doc do you think that should go in?
On Mon, Mar 16, 2015 at 2:45 AM Sven Dowideit notifications@github.com
wrote:

In the PR summary, you document what this feature is useful for - but this
information is not reflected in the documentation changes you've made.

Can you perhaps add the use case example and explanation to the docs too?


Reply to this email directly or view it on GitHub
#11109 (comment).

@ncdc
Contributor
ncdc commented Mar 16, 2015

Actually, could we do that use case doc in a follow-up PR?

@dmcgowan
Member

@SvenDowideit I agree with @ncdc since this is not a feature users must use or get familiar with right away, it might be something many users may want to use later on as we add more "by digest" functionality.

@ncdc
Contributor
ncdc commented Mar 16, 2015

Rebased

@icecrime
Member

Ping @SvenDowideit: are we good?

@SvenDowideit
Collaborator

@icecrime @ncdc at this point, your reader is going to spend more time trying to think up reasons why this digest thing exists, and how and when they'd use it - so IMO, if you add a small statement, they will pass over it much faster than if you leave it as is.

But other than distracting the reader, LGTM.

@ncdc
Contributor
ncdc commented Mar 17, 2015

@icecrime is anything else needed before this can be merged?

@ncdc ncdc Add ability to refer to image by name + digest
Add ability to refer to an image by repository name and digest using the
format repository@digest. Works for pull, push, run, build, and rmi.

Signed-off-by: Andy Goldstein <agoldste@redhat.com>
a2b0c97
@vbatts
Contributor
vbatts commented Mar 17, 2015

looks to me like this is ready to merge

@icecrime icecrime added 4-merge and removed 3-docs-review labels Mar 17, 2015
@icecrime
Member

Cheers

@icecrime icecrime merged commit 0c3c645 into docker:master Mar 17, 2015

2 checks passed

janky Jenkins build Docker-PRs 3487 has succeeded
Details
windows Jenkins build Windows-PRs 571 has succeeded
Details
@vieux vieux removed the 4-merge label Mar 17, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment