-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for referring to images by digest #11109
Conversation
Can you please sign your commits following these rules: https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work The easiest way to do this is to amend the last commit: $ git clone -b "pull-by-digest" git@github.com:ncdc/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~3
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f This will update the existing PR, so you do not need to open a new one. |
Requires distribution/distribution#211 in the v2 registry Test cases are still being written, but I wanted to submit this now given the March 18th code freeze so the maintainers can begin to review it and provide feedback. |
9bd24df
to
ed64cc9
Compare
Can you list some example cli use cases. I see |
Sure thing! Basically any place you potentially want to refer to an image using a tag, you can do it using a digest:
and in a Dockerfile:
|
I also think we probably need to update the responses from the daemon to the client to include the digest (so things like go-dockerclient and other libraries can use it). This would be for push and pull initially, and later on possibly for build and tag as well (if those commands end up generating manifests at some point). And we may want to display the digest to stdout as well. |
Can you define a few of these other changes as well... What happens when deleting all tags? What happens when pulling an image which does not exist locally? |
This is one I wanted input on. This is what's coded now, but I'm not sure what the appropriate desired behavior should be. I could use some help figuring out what makes the most sense here.
I don't like this behavior, and am hoping we can come to a decision on what to do in this situation. We could
That makes the output quite wide, however, and we don't yet support shortened references to digests (TBD in distribution/distribution#194).
Do any of these sound good, or does anyone have other ideas? |
Currently the digests are only computed on the registry side, so then most the Digest fields would be blank unless all images were push or pulled with a v2 registry. For the second case I think I would prefer to just see the Repository and My only worry with the first case is that it could lead to garbage building up in that file. I think removing the Digest link when the image is removed is the right thing to do. But then that relates to the second issue, what does it mean for an image to exist locally without a tag (why does rmi last tag delete an image and not put it in the same state as pulling without a tag?). Will need consensus from @icecrime @jfrazelle @crosbymichael |
In order to be able to write integration tests for this, we'll need distribution/distribution#211 merged, then we'll need to bump the commit in the Dockerfile for the distribution repo so we can spin up a v2 registry with support for pull by digest. |
I'm copying this comment from #10740 here as this is a better place for the discussion: @stevvooe wrote in #10740 (comment) about having a parser that can return an "image object reference". Right now,
type ImageReference struct {
Repository string // or possibly registry.RepositoryInfo instead of string
Tag string
Digest string
} This @stevvooe indicated his desire to see |
@@ -12,6 +12,8 @@ import ( | |||
"github.com/docker/docker/utils" | |||
) | |||
|
|||
const DockerDigestHeader = "Docker-Digest" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would we prefer this as Docker-Content-Digest
?
cc @ncdc
@ncdc This is really up to the use case. In the |
+1 For the As for the output format, I think that changing the default output of
The output is: Image ID, All local tags for the given ID, Digest, RepoName |
I agree that changing the output of |
I've done a bit of work attempting to replace While it would be great to have, creating |
@ncdc while I think the For now lets get this ready for the 1.6 window. |
Is push by digest intended to be part of this PR? Along the same lines may also be tag by digest. Currently to push an image I pulled by digest, I must tag by image id then push using that tag. |
Push by digest is supposed to be part of this PR. If it's not working, I might have overlooked testing that specific feature. I don't think it's critical to our workflow (OpenShift), but I think it makes sense to include it. Are you saying that |
I don't think push by digest should be part of this PR and rather excluded until a later version which supports schema version 2 of the manifest. The current version of the manifest requires a tag to be set. Since the digest represents a manifest which contained a tag, that tag is what should be used for the push. However pull by digest currently does not use the manifest tag (perhaps it should) and it does not store the manifest. Since the manifest is not stored, pushing by digest will always recreate a new manifest and tars to push, causing a new digest to be created for the image id. @jlhawn is also working on a new implementation of storage of layers and manifest which would enable deterministic pushing of layers and manifests. It is probably best to wait until a new storage backend to support this. The tag command should be supported if the digests are going to stick around for |
I'm happy to omit push by digest for now. FYI, pushing the same manifest content multiple times should result in the same digest, as the unsigned portion of the (recreated) manifest should remain the same if nothing else has changed.... right? That's what I'm seeing locally, at least.
I'm not against adding support for tagging a digest to a new tag, but I'm not sure how |
We are not guaranteeing in this version of the manifest that the unsigned portion will not change. It shouldn't but there have been issues with using tarsum. I just meant that if the digest is intended to be used for commands after pull, such as |
We can certainly do that, but, at least for OpenShift, we plan on passing name@digest to |
@dmcgowan I'm able to |
In the PR summary, you document what this feature is useful for - but this information is not reflected in the documentation changes you've made. Can you perhaps add the use case example and explanation to the docs too? |
I can do that. Which doc do you think that should go in?
|
Actually, could we do that use case doc in a follow-up PR? |
@SvenDowideit I agree with @ncdc since this is not a feature users must use or get familiar with right away, it might be something many users may want to use later on as we add more "by digest" functionality. |
Rebased |
Ping @SvenDowideit: are we good? |
@icecrime @ncdc at this point, your reader is going to spend more time trying to think up reasons why this But other than distracting the reader, LGTM. |
@icecrime is anything else needed before this can be merged? |
Add ability to refer to an image by repository name and digest using the format repository@digest. Works for pull, push, run, build, and rmi. Signed-off-by: Andy Goldstein <agoldste@redhat.com>
looks to me like this is ready to merge |
Add support for referring to images by digest
Overview
When I create a container, I may specify an image such as
mysql:latest
. When the image is pulled,latest
is resolved to a particular image at that point in time. If I later want to add more containers (e.g. possible read slaves in the MySQL case), all the new containers must use the exact same image as my first container. Using a tag isn't sufficient as the tag can be updated to point at a different image. We need a way to refer to images using immutable identifiers.With the v2 image format, image manifests have content-addressable, immutable digests. The v2 registry supports retrieving manifests by digest; this pull request adds the corresponding support to the Docker Core.
Digest reference format
We'll need to provide a means to reference an image by its digest. I'd like to propose the following format:
Supported commands
We'll need to make sure the following commands continue to work as they currently do, as well as with an optional digest:
docker build
docker pull
docker create
docker run
docker rmi
This list should eventually be comprehensive; anywhere you can refer to an image, you should be able to do so by digest. For the time being, the commands listed above should be sufficient for the use case listed in the overview.
docker images
If you pull an image from a v2 registry, the registry provides the image manifest's digest as a response header. If you pull by tag, you'll have both the tag and the digest. If you pull only by digest, you won't have the tag information. 1 question to resolve is how to display images, tags, digests, and v1 image IDs in the
docker images
command.It is possible (likely?) that a variety of code exists to scrape the output of
docker images
. If we change the format, e.g. by adding a new column, or overloading an existing column to sometimes show a v1 image ID vs a digest, we will probably break whatever code is out there doing the scraping. We need to determine how critical it is to retain the existing column and data formats in thedocker images
command.I have the following ideas regarding this output:
repository
,tag
,digest
, andimage id
repository
, for tag,digest
, andimage id
docker images
questionsDIGEST
column?docker images
- you won't see the image you just pulledQuestions
What about v1 registry support?
The v1 registry won't support this feature.
If I create an image locally via
docker tag
ordocker commit
, can I refer to it byname@digest
?As proposed in distribution/distribution#46, the registry is responsible for determining an image's digest and assigning it to the image. For an image that has not yet been pushed to a v2 registry, it may not be possible to refer to it by
name@digest
. This is unlikely to be a significant issue, as the use case forname@digest
is consistent deployments using images pulled from registries. Or, if the community thinks this should be supported, we can revisit what component(s) are responsible for calculating digests.Additional information
See #10740 for more backstory.
Todo: