Add support for referring to images by digest

[jlhawn]

Supersedes #11109 by @ncdc with implementation of #11109 (comment)

Overview

When I create a container, I may specify an image such as mysql:latest. When the image is pulled, latest is resolved to a particular image at that point in time. If I later want to add more containers (e.g. possible read slaves in the MySQL case), all the new containers must use the exact same image as my first container. Using a tag isn't sufficient as the tag can be updated to point at a different image. We need a way to refer to images using immutable identifiers.

With the v2 image format, image manifests have content-addressable, immutable digests. The v2 registry supports retrieving manifests by digest; this pull request adds the corresponding support to the Docker Core.

Digest reference format

We'll need to provide a means to reference an image by its digest. I'd like to propose the following format:

namespace/repository@digest

Supported commands

We'll need to make sure the following commands continue to work as they currently do, as well as with an optional digest:

• docker build
• docker pull
• docker create
• docker run
• docker rmi

This list should eventually be comprehensive; anywhere you can refer to an image, you should be able to do so by digest. For the time being, the commands listed above should be sufficient for the use case listed in the overview.

docker images

If you pull an image from a v2 registry, the registry provides the image manifest's digest as a response header. If you pull by tag, you'll have both the tag and the digest. If you pull only by digest, you won't have the tag information. 1 question to resolve is how to display images, tags, digests, and v1 image IDs in the docker images command.

It is possible (likely?) that a variety of code exists to scrape the output of docker images. If we change the format, e.g. by adding a new column, or overloading an existing column to sometimes show a v1 image ID vs a digest, we will probably break whatever code is out there doing the scraping. We need to determine how critical it is to retain the existing column and data formats in the docker images command.

I have the following ideas regarding this output:

• if an image was pulled by tag, and the registry it's pulled from includes the image's digest, display repository, tag, digest, and image id
• if an image was pulled by digest, display repository, for tag, digest, and image id

docker images questions

• should we add a new DIGEST column?
• if yes, should we always display it, or only display it if you indicate that you want to see digests (e.g. via a filter)
  • there can be confusion if you pull an image by digest and then run docker images - you won't see the image you just pulled
• should we display the digest instead of the v1 image ID (or some other value)?

Questions

What about v1 registry support?
The v1 registry won't support this feature.

If I create an image locally via docker tag or docker commit, can I refer to it by name@digest?
As proposed in distribution/distribution#46, the registry is responsible for determining an image's digest and assigning it to the image. For an image that has not yet been pushed to a v2 registry, it may not be possible to refer to it by name@digest. This is unlikely to be a significant issue, as the use case for name@digest is consistent deployments using images pulled from registries. Or, if the community thinks this should be supported, we can revisit what component(s) are responsible for calculating digests.

Additional information

See #10740 for more backstory.

Todo:

• Test cases
• Update documentation

[GordonTheTurtle]

These files are not properly gofmt'd:
api/client/commands.go
Please reformat the above files using gofmt -s -w and amend to the commit the result.

[jlhawn]

@GordonTheTurtle I don't know what you're talking about. Is it a specific commit that you're complaining about because the tip of the branch seems fine.

@jfrazelle ^ is this guy serious?

[ncdc]

@jlhawn it looks like you changed the behavior so that it no longer stores the digest->id reference when pushing or pulling by tag. I've restored this in an additional commit: ncdc@a6df96e.