New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add manifest command #138

Merged
merged 4 commits into from Jan 18, 2018

Conversation

@clnperez
Contributor

clnperez commented May 30, 2017

Enable inspection (aka "shallow pull") of images' manifest info, and also the creation of manifest lists (aka "fat manifests").

The workflow for creating a manifest list will be:

docker manifest create new-list-ref-name image-ref [image-ref...]
docker manifest annotate new-list-ref-name image-ref --os linux --arch arm
docker manifest push new-list-ref-name

There is also a manifest inspect command to allow for a "shallow pull"
of an image's manifest: docker manifest inspect manifest-or-manifest_list.

To be more in line with the existing external manifest tool, there is
also a -v option for inspect that will show information depending on
what the reference maps to (list or single manifest).

Signed-off-by: Christy Norman Perez christy@linux.vnet.ibm.com

- What I did

- How I did it

- How to verify it

- Description for the changelog

See moby/moby#27455 for history.

TODO:

  • move/refactor tests from original pr to this
@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez
Contributor

clnperez commented May 30, 2017

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin May 30, 2017

Collaborator

Thanks for moving this PR over to the docker/cli repo!

This PR is quite large! It would be much easier to review if it could be split up and submitted incrementally. Would it be possible to include just one or two sub commands at first? The YAML config I think could be added later on.

We'd like to keep a separation between the cli/command packages (which are responsible for flag parsing, and command routing) and "client logic". Most client commands don't have much logic, but you can see this separation between the cli/compose and cli/command/stack packages. fetch.go, fetch_v1.go, and fetch_v2.go should be moved into some other package (maybe cli/manfiest).

Collaborator

dnephin commented May 30, 2017

Thanks for moving this PR over to the docker/cli repo!

This PR is quite large! It would be much easier to review if it could be split up and submitted incrementally. Would it be possible to include just one or two sub commands at first? The YAML config I think could be added later on.

We'd like to keep a separation between the cli/command packages (which are responsible for flag parsing, and command routing) and "client logic". Most client commands don't have much logic, but you can see this separation between the cli/compose and cli/command/stack packages. fetch.go, fetch_v1.go, and fetch_v2.go should be moved into some other package (maybe cli/manfiest).

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 30, 2017

Contributor

@dnephin It would be a lot of work to split things back out. I realize it would be easier to review and I of course don't want to make reviewers' lives harder -- but at the same time, I would appreciate some compromise. To use your yaml suggestion as an example, I was asked to remove the yaml func, then to add it back in, so I've already been through this twice with the yaml feature alone. I'm not trying to be stubborn -- just asking for a little mercy where it can be given (without sacrificing project integrity, of course).

As for including only a few of the sub-commands, none of the ones there make sense without all the others, except maybe inspect. But, I think that's just silly to take out and put back in. 😹

As for your other point about splitting some of it out into another package, I think that makes sense and I can look into it.

Contributor

clnperez commented May 30, 2017

@dnephin It would be a lot of work to split things back out. I realize it would be easier to review and I of course don't want to make reviewers' lives harder -- but at the same time, I would appreciate some compromise. To use your yaml suggestion as an example, I was asked to remove the yaml func, then to add it back in, so I've already been through this twice with the yaml feature alone. I'm not trying to be stubborn -- just asking for a little mercy where it can be given (without sacrificing project integrity, of course).

As for including only a few of the sub-commands, none of the ones there make sense without all the others, except maybe inspect. But, I think that's just silly to take out and put back in. 😹

As for your other point about splitting some of it out into another package, I think that makes sense and I can look into it.

@clnperez clnperez referenced this pull request May 30, 2017

Closed

Add manifest command to docker cli #27455

3 of 6 tasks complete
Show outdated Hide outdated cli/command/manifest/annotate.go
Show outdated Hide outdated cli/command/manifest/annotate.go
Show outdated Hide outdated cli/command/manifest/annotate.go
Show outdated Hide outdated cli/command/manifest/fetch.go
Show outdated Hide outdated cli/command/manifest/fetch_v2.go
Show outdated Hide outdated cli/command/manifest/inspect.go
Show outdated Hide outdated cli/command/manifest/inspect.go
Show outdated Hide outdated cli/command/manifest/inspect.go
Show outdated Hide outdated cli/command/manifest/inspect.go
Show outdated Hide outdated cli/command/manifest/inspect.go
@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Jun 2, 2017

Codecov Report

Merging #138 into master will increase coverage by 0.13%.
The diff coverage is 67.92%.

@@            Coverage Diff             @@
##           master     #138      +/-   ##
==========================================
+ Coverage   51.23%   51.36%   +0.13%     
==========================================
  Files         237      244       +7     
  Lines       15399    15762     +363     
==========================================
+ Hits         7889     8096     +207     
- Misses       7008     7121     +113     
- Partials      502      545      +43

codecov-io commented Jun 2, 2017

Codecov Report

Merging #138 into master will increase coverage by 0.13%.
The diff coverage is 67.92%.

@@            Coverage Diff             @@
##           master     #138      +/-   ##
==========================================
+ Coverage   51.23%   51.36%   +0.13%     
==========================================
  Files         237      244       +7     
  Lines       15399    15762     +363     
==========================================
+ Hits         7889     8096     +207     
- Misses       7008     7121     +113     
- Partials      502      545      +43
@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer Jun 14, 2017

Contributor

I just found out that I can use docker manifest create and docker manifest push to remote "tag" an existing image to another one.

Use case: Moving an image from one stage to another (after successful tests on different machines) without pulling and pushing the whole image. And probably able to retag Windows images as well on a Linux machine (where you can't pull and push Windows images).

PS: I tried that with manifest-tool 0.5.0, but get an error "You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion."

So +1 👍 having the "docker manifest" command.

Oh, this only works if the source manifest only has one platform, for a multiarch manifest as source the same error occurs:

$ docker manifest create stefanscherer/winspector:staging stefanscherer/winspector:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.
Contributor

StefanScherer commented Jun 14, 2017

I just found out that I can use docker manifest create and docker manifest push to remote "tag" an existing image to another one.

Use case: Moving an image from one stage to another (after successful tests on different machines) without pulling and pushing the whole image. And probably able to retag Windows images as well on a Linux machine (where you can't pull and push Windows images).

PS: I tried that with manifest-tool 0.5.0, but get an error "You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion."

So +1 👍 having the "docker manifest" command.

Oh, this only works if the source manifest only has one platform, for a multiarch manifest as source the same error occurs:

$ docker manifest create stefanscherer/winspector:staging stefanscherer/winspector:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.
@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jun 14, 2017

Contributor

@StefanScherer That looks like a bug. Perhaps, if one creates a manifest list from an existing manifest list, it should clone it, rather than trying to reference it.

Contributor

stevvooe commented Jun 14, 2017

@StefanScherer That looks like a bug. Perhaps, if one creates a manifest list from an existing manifest list, it should clone it, rather than trying to reference it.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jun 14, 2017

Contributor

@clnperez Are there a few examples of the manifest inspect output? I ask because the output of these commands tend to become what people think of when we talk about the data structure. This can be problematic, because people will look at that and say "Just add x", when that is actually impossible, because the internal data structure is misrepresented.

Contributor

stevvooe commented Jun 14, 2017

@clnperez Are there a few examples of the manifest inspect output? I ask because the output of these commands tend to become what people think of when we talk about the data structure. This can be problematic, because people will look at that and say "Just add x", when that is actually impossible, because the internal data structure is misrepresented.

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Jun 14, 2017

Member

I think what @StefanScherer is looking for is more like a "manifest clone" or what might be like a docker rtag command (instead of tag locally, tag remotely) if the only difference is a tag name.

Instead, using manifest create is thinking that you want to create a manifest list of of the first entry, which is already a manifest list, hence the error message. You end up needing a new format or command to specify something more along the lines of "duplicate this same manifest in the remote registry, but with a different tag". I just added a PR for that in manifest-tool by allowing a set of "additional tags" to be provided, but a more general solution would be a remote tag operation which assumes all references and blob mounts are already handled (which would be the case for an existing manifest reference in a repo).

Member

estesp commented Jun 14, 2017

I think what @StefanScherer is looking for is more like a "manifest clone" or what might be like a docker rtag command (instead of tag locally, tag remotely) if the only difference is a tag name.

Instead, using manifest create is thinking that you want to create a manifest list of of the first entry, which is already a manifest list, hence the error message. You end up needing a new format or command to specify something more along the lines of "duplicate this same manifest in the remote registry, but with a different tag". I just added a PR for that in manifest-tool by allowing a set of "additional tags" to be provided, but a more general solution would be a remote tag operation which assumes all references and blob mounts are already handled (which would be the case for an existing manifest reference in a repo).

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jun 14, 2017

Contributor

@stevvooe here's what I had pasted in the original PR: moby/moby#27455 (comment)

Since then I changed the flag from -p for platform to -v for verbose so that it fits with both manifests and manifest lists. I extended the additional/verbose output to manifest lists so that users could also output layers (by way of the individual image's json, img.CanonicalJSON) since people seemed to be comparing layers from the previous tool's output.

I'm not aware of the the use-case, though. @StefanScherer, @estesp, @tianon, et al., do you think displaying layers of the image referenced by the manifest list is useful?

Here's an image's manifest, and it's image data (including layers). It's just the last one printed (which happens to be the arm image's info) when I run
docker-linux-amd64 manifest inspect -v clnperez/hello-world

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1462,
        "digest": "sha256:d40384c3f8619d0976e52dd6aa13e6037e16698205a95e87d5d468dc12753630"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 986,
            "digest": "sha256:a0691bf12e4effeae0f10e3e0f38e67ab16315ac870f2b35b8500ead5b5904e1"
        }
    ]
}

@stevvooe If you think having the -v is bad and will cause the scenario you described, and people think that the layers output is useful, I can remove this flag for now and we can figure out something better later after the PR is in.

Contributor

clnperez commented Jun 14, 2017

@stevvooe here's what I had pasted in the original PR: moby/moby#27455 (comment)

Since then I changed the flag from -p for platform to -v for verbose so that it fits with both manifests and manifest lists. I extended the additional/verbose output to manifest lists so that users could also output layers (by way of the individual image's json, img.CanonicalJSON) since people seemed to be comparing layers from the previous tool's output.

I'm not aware of the the use-case, though. @StefanScherer, @estesp, @tianon, et al., do you think displaying layers of the image referenced by the manifest list is useful?

Here's an image's manifest, and it's image data (including layers). It's just the last one printed (which happens to be the arm image's info) when I run
docker-linux-amd64 manifest inspect -v clnperez/hello-world

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1462,
        "digest": "sha256:d40384c3f8619d0976e52dd6aa13e6037e16698205a95e87d5d468dc12753630"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 986,
            "digest": "sha256:a0691bf12e4effeae0f10e3e0f38e67ab16315ac870f2b35b8500ead5b5904e1"
        }
    ]
}

@stevvooe If you think having the -v is bad and will cause the scenario you described, and people think that the layers output is useful, I can remove this flag for now and we can figure out something better later after the PR is in.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jun 14, 2017

Contributor

@StefanScherer can you paste what you were doing that Steven said sounds like a bug? I think I'm misunderstanding what you did b/c it sounds to me like you just made a manifest list with only one entry in it, which is fine. But I don't see why Phil's tool wouldn't allow that.

Contributor

clnperez commented Jun 14, 2017

@StefanScherer can you paste what you were doing that Steven said sounds like a bug? I think I'm misunderstanding what you did b/c it sounds to me like you just made a manifest list with only one entry in it, which is fine. But I don't see why Phil's tool wouldn't allow that.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jun 14, 2017

Contributor

@clnperez Do we ever print out ImgManifestInspect? That seems useful as display. So, as I understand, we print that by default and the regular manifest with -v?

Contributor

stevvooe commented Jun 14, 2017

@clnperez Do we ever print out ImgManifestInspect? That seems useful as display. So, as I understand, we print that by default and the regular manifest with -v?

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jun 15, 2017

Contributor

@stevvooe No, we don't print that out. That's the "franken-manifest" you thought was a bad idea before. If you've reconsidered and you think it's useful I could easily put that back, and only print the manifest struct with the -v flag. Right now I print:

docker manifest inspect img-ref
manifest object

docker manifest inspect -v img-ref
manifest object
manifestdescriptor

docker manifest inspect manifest-list
DeserializedManifestList

docker manifest inspect -v manifest-list
DeserialzedManifestList
[manifest obj1, manifest obj2...]

Contributor

clnperez commented Jun 15, 2017

@stevvooe No, we don't print that out. That's the "franken-manifest" you thought was a bad idea before. If you've reconsidered and you think it's useful I could easily put that back, and only print the manifest struct with the -v flag. Right now I print:

docker manifest inspect img-ref
manifest object

docker manifest inspect -v img-ref
manifest object
manifestdescriptor

docker manifest inspect manifest-list
DeserializedManifestList

docker manifest inspect -v manifest-list
DeserialzedManifestList
[manifest obj1, manifest obj2...]

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jun 15, 2017

Contributor

@clnperez If you think it is useful to show the "franken manifest" and make it clear that it is not the manifest but a summary of data structures, then by all means, let's have something along those lines. Either way, this is looking good.

@dnephin What are the real blockers for getting this in as experimental, at least?

Contributor

stevvooe commented Jun 15, 2017

@clnperez If you think it is useful to show the "franken manifest" and make it clear that it is not the manifest but a summary of data structures, then by all means, let's have something along those lines. Either way, this is looking good.

@dnephin What are the real blockers for getting this in as experimental, at least?

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer Jun 15, 2017

Contributor

@clnperez As Phil mentioned I'm more searching for a way to remote copy/clone a tag or manifest without pulling the image (or images which is eg. not possible for Windows images if running through a Linux Docker host).

My steps to reproduce the situation is this.

Trying to create a manifest from a manifest list which only has one platform in it works:

$ ~/Downloads/docker-darwin-amd64 manifest create stefanscherer/ngrokd:prod stefanscherer/ngrokd:staging
INFO[0000] Retrieving digests of images...              

This also can be pushed afterwards.

Trying to create a manifest from a multiarch manifest list does not work:

$ docker manifest create stefanscherer/winspector:prod stefanscherer/whoami:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.

So a "copy" or "clone" command would describe the step better than a "create" command.

Contributor

StefanScherer commented Jun 15, 2017

@clnperez As Phil mentioned I'm more searching for a way to remote copy/clone a tag or manifest without pulling the image (or images which is eg. not possible for Windows images if running through a Linux Docker host).

My steps to reproduce the situation is this.

Trying to create a manifest from a manifest list which only has one platform in it works:

$ ~/Downloads/docker-darwin-amd64 manifest create stefanscherer/ngrokd:prod stefanscherer/ngrokd:staging
INFO[0000] Retrieving digests of images...              

This also can be pushed afterwards.

Trying to create a manifest from a multiarch manifest list does not work:

$ docker manifest create stefanscherer/winspector:prod stefanscherer/whoami:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.

So a "copy" or "clone" command would describe the step better than a "create" command.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jun 15, 2017

Contributor

Thanks for the clarification @StefanScherer. That's not a bug so much as an unintended use-case. ;) I feel like we should start a list of "Features to add once this PR is in." I'm seeing a lot of cool things go into @estesp's manifest tool lately (which are exciting, b/c that means we're getting closer to multi-arch usage in the wild).

Contributor

clnperez commented Jun 15, 2017

Thanks for the clarification @StefanScherer. That's not a bug so much as an unintended use-case. ;) I feel like we should start a list of "Features to add once this PR is in." I'm seeing a lot of cool things go into @estesp's manifest tool lately (which are exciting, b/c that means we're getting closer to multi-arch usage in the wild).

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Jun 16, 2017

Collaborator

What are the real blockers for getting this in as experimental, at least?

The package split up from my last comment, and time to review. This is a lot of code. It's going to take a long time to review. Anything we can do to split this into smaller PRs will reduce the time it takes to get it merged into experimental.

I'd like to suggest splitting out a PR for just the inspect subcommand. That would be much faster to review.

Also I guess fixing the CI failures, and adding unit tests for the new code.

Collaborator

dnephin commented Jun 16, 2017

What are the real blockers for getting this in as experimental, at least?

The package split up from my last comment, and time to review. This is a lot of code. It's going to take a long time to review. Anything we can do to split this into smaller PRs will reduce the time it takes to get it merged into experimental.

I'd like to suggest splitting out a PR for just the inspect subcommand. That would be much faster to review.

Also I guess fixing the CI failures, and adding unit tests for the new code.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jun 16, 2017

Contributor

@dnephin The PR has been open for months now. This needs to move forward.

Do you have any actual objections to the design?

Contributor

stevvooe commented Jun 16, 2017

@dnephin The PR has been open for months now. This needs to move forward.

Do you have any actual objections to the design?

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Jun 16, 2017

Collaborator

It has been open for so long because it is too large to review.
I can't evaluate the design in the current form.

Collaborator

dnephin commented Jun 16, 2017

It has been open for so long because it is too large to review.
I can't evaluate the design in the current form.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jun 16, 2017

Contributor

@dnephin That is ridiculous. Where are our other maintainers?

Contributor

stevvooe commented Jun 16, 2017

@dnephin That is ridiculous. Where are our other maintainers?

@vdemeester

This comment has been minimized.

Show comment
Hide comment
@vdemeester

vdemeester Jun 16, 2017

Member

@clnperez could you split into 2 commits (one for the vendoring) ? (to ease the review 👼)

Member

vdemeester commented Jun 16, 2017

@clnperez could you split into 2 commits (one for the vendoring) ? (to ease the review 👼)

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jan 18, 2018

Contributor

/me faints

Thanks @vdemeester. And @dnephin. And everyone for all the help for the past year. ❤️

Contributor

clnperez commented Jan 18, 2018

/me faints

Thanks @vdemeester. And @dnephin. And everyone for all the help for the past year. ❤️

@deitch

This comment has been minimized.

Show comment
Hide comment
@deitch

deitch Jan 19, 2018

Fabulous! What version of docker will it roll into?

deitch commented Jan 19, 2018

Fabulous! What version of docker will it roll into?

@LelandSindt

This comment has been minimized.

Show comment
Hide comment
@LelandSindt

LelandSindt commented Jan 19, 2018

@deitch looks like it will be rolled into 18.02.0

https://github.com/docker/cli/milestone/9?closed=1

@AceHack

This comment has been minimized.

Show comment
Hide comment
@AceHack

AceHack Jan 31, 2018

Are there plans to support the yml like file format from manifest-tool?

AceHack commented Jan 31, 2018

Are there plans to support the yml like file format from manifest-tool?

@AceHack

This comment has been minimized.

Show comment
Hide comment
@AceHack

AceHack Jan 31, 2018

Also is there any plan to be able to update manifests one at a time? For instance, I have builds for windows and Linux and they do not have any dependencies on each other so when one finishes I just want it to go create/update it's own subtag under the main tag. Each build process can be isolated in this way.

AceHack commented Jan 31, 2018

Also is there any plan to be able to update manifests one at a time? For instance, I have builds for windows and Linux and they do not have any dependencies on each other so when one finishes I just want it to go create/update it's own subtag under the main tag. Each build process can be isolated in this way.

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Jan 31, 2018

Member

@AceHack

Are there plans to support the yml like file format from manifest-tool?

yes, some features were removed from the initial PR to make it more manageable for review. As soon as @clnperez recovers from the feat of this PR, I think there are plans to create PRs to add those capabilities back in.

Also is there any plan to be able to update manifests one at a time?

A manifest list is one entity in the registry; there is no way to only modify one part of an existing entity; however, continued (re-)pushes with "sub"/architecture builds that are complete would be one way to solve this. This means your repo name/tag reference will be changing as platform entries are added/removed, which could cause downstream image user churn. As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

Member

estesp commented Jan 31, 2018

@AceHack

Are there plans to support the yml like file format from manifest-tool?

yes, some features were removed from the initial PR to make it more manageable for review. As soon as @clnperez recovers from the feat of this PR, I think there are plans to create PRs to add those capabilities back in.

Also is there any plan to be able to update manifests one at a time?

A manifest list is one entity in the registry; there is no way to only modify one part of an existing entity; however, continued (re-)pushes with "sub"/architecture builds that are complete would be one way to solve this. This means your repo name/tag reference will be changing as platform entries are added/removed, which could cause downstream image user churn. As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

@AceHack

This comment has been minimized.

Show comment
Hide comment
@AceHack

AceHack Jan 31, 2018

@estesp thanks so much for your quick response. So in your 2nd comment on being able to run one at a time. I'm okay with the churn so let me ask a scenario.

If I have 2 independent build pipelines that do the following.

----Build Pipeline 1-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-1}
docker manifest annotate {multi-arch-tag} {os-specific-tag-1} --os {os-1} --arch {arch-1}
docker manifest push --purge {multi-arch-tag}

----Build Pipeline 2-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-2}
docker manifest annotate {multi-arch-tag} {os-specific-tag-2} --os {os-2} --arch {arch-2}
docker manifest push --purge {multi-arch-tag}

Is this last one wins? Where the {multi-arch-tag} has one subtag either {os-specific-tag-1} or {os-specific-tag-2}, basically the last build to run wins?
Or is it additive so that you end up with both the {os-specific-tag-1} and {os-specific-tag-2} under the main {multi-arch-tag}?

Again, thanks so much.

AceHack commented Jan 31, 2018

@estesp thanks so much for your quick response. So in your 2nd comment on being able to run one at a time. I'm okay with the churn so let me ask a scenario.

If I have 2 independent build pipelines that do the following.

----Build Pipeline 1-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-1}
docker manifest annotate {multi-arch-tag} {os-specific-tag-1} --os {os-1} --arch {arch-1}
docker manifest push --purge {multi-arch-tag}

----Build Pipeline 2-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-2}
docker manifest annotate {multi-arch-tag} {os-specific-tag-2} --os {os-2} --arch {arch-2}
docker manifest push --purge {multi-arch-tag}

Is this last one wins? Where the {multi-arch-tag} has one subtag either {os-specific-tag-1} or {os-specific-tag-2}, basically the last build to run wins?
Or is it additive so that you end up with both the {os-specific-tag-1} and {os-specific-tag-2} under the main {multi-arch-tag}?

Again, thanks so much.

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Jan 31, 2018

Member

As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

Just to be clear, the root issue isn't actually solved -- still crops up from time to time because we only have a hacky workaround to help it happen less often currently. 😅 (docker-library/official-images#3835)

Member

tianon commented Jan 31, 2018

As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

Just to be clear, the root issue isn't actually solved -- still crops up from time to time because we only have a hacky workaround to help it happen less often currently. 😅 (docker-library/official-images#3835)

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jan 31, 2018

Contributor

@AceHack I have the yaml re-add on my todo for next week hopefully.

It would be nice if we could find a way to only update one part of a manifest list on a registry (maybe by pulling, copying, and repushing with original bits and updated bits).

Contributor

clnperez commented Jan 31, 2018

@AceHack I have the yaml re-add on my todo for next week hopefully.

It would be nice if we could find a way to only update one part of a manifest list on a registry (maybe by pulling, copying, and repushing with original bits and updated bits).

@AceHack

This comment has been minimized.

Show comment
Hide comment
@AceHack

AceHack Feb 1, 2018

It would also be nice to use some sort of etag i.e. optimistic concurrency concept so in the unlikely case that both push at the same time there will be no concurrency funny business. One will win and the other will just get the etag failure and retry. Thanks.

AceHack commented Feb 1, 2018

It would also be nice to use some sort of etag i.e. optimistic concurrency concept so in the unlikely case that both push at the same time there will be no concurrency funny business. One will win and the other will just get the etag failure and retry. Thanks.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Feb 8, 2018

Contributor

Here's the PR for the yaml re-add: #866

Contributor

clnperez commented Feb 8, 2018

Here's the PR for the yaml re-add: #866

@fkrull fkrull referenced this pull request Feb 12, 2018

Merged

Docker Images #3029

@khatribharat

This comment has been minimized.

Show comment
Hide comment
@khatribharat

khatribharat Feb 17, 2018

Docker CE Edge documentation for manifest command available here.

khatribharat commented Feb 17, 2018

Docker CE Edge documentation for manifest command available here.

@agronholm

This comment has been minimized.

Show comment
Hide comment
@agronholm

agronholm Feb 18, 2018

I tried to create a manifest list but failed utterly. I created two images, say registry.example.org/example-amd64:staging and registry.example.org/example-arm:staging and then tried to created a manifest list using docker manifest create registry.example.org/example:staging registry.example.org/example-amd64:staging registry.example.org/example-arm:staging but got this error: no such manifest: registry.example.org/example-amd64:staging

I was under the impression that every image also has a manifest. If this is not true, how do I create the manifest for the image? This part is missing from the documentation.

agronholm commented Feb 18, 2018

I tried to create a manifest list but failed utterly. I created two images, say registry.example.org/example-amd64:staging and registry.example.org/example-arm:staging and then tried to created a manifest list using docker manifest create registry.example.org/example:staging registry.example.org/example-amd64:staging registry.example.org/example-arm:staging but got this error: no such manifest: registry.example.org/example-amd64:staging

I was under the impression that every image also has a manifest. If this is not true, how do I create the manifest for the image? This part is missing from the documentation.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Feb 18, 2018

Contributor

@agronholm would your registry happen to not be using tls? if the command can't find the manifest on a registry that meets all the default requirements, then it will just tell you one doesn't exist. if that's the case, you can use the --insecure flag on create & push.

Contributor

clnperez commented Feb 18, 2018

@agronholm would your registry happen to not be using tls? if the command can't find the manifest on a registry that meets all the default requirements, then it will just tell you one doesn't exist. if that's the case, you can use the --insecure flag on create & push.

@agronholm

This comment has been minimized.

Show comment
Hide comment
@agronholm

agronholm Feb 18, 2018

No, it is using TLS. Also, the documentation did not give me any indication that manifest create accesses the registry. I thought it was purely a local operation. That said, I did have the relevant images pushed to the registry before this. Apparently this is a requirement?

agronholm commented Feb 18, 2018

No, it is using TLS. Also, the documentation did not give me any indication that manifest create accesses the registry. I thought it was purely a local operation. That said, I did have the relevant images pushed to the registry before this. Apparently this is a requirement?

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Feb 19, 2018

Contributor

Ah, yes. It sounds like there needs to be some clarification around that. It's mentioned in the insecure registries section, but if you aren't working with an insecure registry you're not going to read that section. :)

Contributor

clnperez commented Feb 19, 2018

Ah, yes. It sounds like there needs to be some clarification around that. It's mentioned in the insecure registries section, but if you aren't working with an insecure registry you're not going to read that section. :)

@agronholm

This comment has been minimized.

Show comment
Hide comment
@agronholm

agronholm Feb 19, 2018

Alright, so since I am not using an insecure registry, how can I figure out why it tells me there is no such manifest?

agronholm commented Feb 19, 2018

Alright, so since I am not using an insecure registry, how can I figure out why it tells me there is no such manifest?

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Feb 19, 2018

Contributor

I think I may have misread your previous comment:

That said, I did have the relevant images pushed to the registry before this.
Apparently this is a requirement?

The language sounded like you didn't have them pushed. Do you? That is a requirement.

Contributor

clnperez commented Feb 19, 2018

I think I may have misread your previous comment:

That said, I did have the relevant images pushed to the registry before this.
Apparently this is a requirement?

The language sounded like you didn't have them pushed. Do you? That is a requirement.

@agronholm

This comment has been minimized.

Show comment
Hide comment
@agronholm

agronholm Feb 20, 2018

The language sounded like you didn't have them pushed. Do you? That is a requirement.

I did try that but it made no difference.

agronholm commented Feb 20, 2018

The language sounded like you didn't have them pushed. Do you? That is a requirement.

I did try that but it made no difference.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Feb 20, 2018

Contributor

Can you try with the -D flag and see if you get any new info?

Contributor

clnperez commented Feb 20, 2018

Can you try with the -D flag and see if you get any new info?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment