Add manifest command

[clnperez]

Enable inspection (aka "shallow pull") of images' manifest info, and also the creation of manifest lists (aka "fat manifests").

The workflow for creating a manifest list will be:

docker manifest create new-list-ref-name image-ref [image-ref...]
docker manifest annotate new-list-ref-name image-ref --os linux --arch arm
docker manifest push new-list-ref-name

There is also a manifest inspect command to allow for a "shallow pull"
of an image's manifest: docker manifest inspect manifest-or-manifest_list.

To be more in line with the existing external manifest tool, there is
also a -v option for inspect that will show information depending on
what the reference maps to (list or single manifest).

Signed-off-by: Christy Norman Perez christy@linux.vnet.ibm.com

- What I did

- How I did it

- How to verify it

- Description for the changelog

See moby/moby#27455 for history.

TODO:

• move/refactor tests from original pr to this

[clnperez]

@stevvooe @estesp FYI

[dnephin]

Thanks for moving this PR over to the docker/cli repo!

This PR is quite large! It would be much easier to review if it could be split up and submitted incrementally. Would it be possible to include just one or two sub commands at first? The YAML config I think could be added later on.

We'd like to keep a separation between the cli/command packages (which are responsible for flag parsing, and command routing) and "client logic". Most client commands don't have much logic, but you can see this separation between the cli/compose and cli/command/stack packages. fetch.go, fetch_v1.go, and fetch_v2.go should be moved into some other package (maybe cli/manfiest).

[clnperez]

@dnephin It would be a lot of work to split things back out. I realize it would be easier to review and I of course don't want to make reviewers' lives harder -- but at the same time, I would appreciate some compromise. To use your yaml suggestion as an example, I was asked to remove the yaml func, then to add it back in, so I've already been through this twice with the yaml feature alone. I'm not trying to be stubborn -- just asking for a little mercy where it can be given (without sacrificing project integrity, of course).

As for including only a few of the sub-commands, none of the ones there make sense without all the others, except maybe inspect. But, I think that's just silly to take out and put back in. 😹

As for your other point about splitting some of it out into another package, I think that makes sense and I can look into it.

[aaronlehmann]

cpu-features?

[clnperez]

Ah, sure.

[aaronlehmann]

os-features?

[clnperez]

Yep, same. Thanks.

[aaronlehmann]

Suggest using Wrap from github.com/pkg/errors.

[aaronlehmann]

Manifest lists don't support v1 registries, so I'm unsure what v1ManifestFetcher is useful for. Even if v1 registries are supported in some way, I would recommend against adding support for them in a new feature.

[clnperez]

Maybe not clear in the initial comment, but, this subcommand isn't just for manifest lists. It also allows you to do docker manifest inspect [blah] of an image name that could be living in a v1 registry. This is so that users can do the "shallow pull" of an image that several people had requested in earlier issues

[stevvooe]

@clnperez How do v1 registries have manifests? Do you have an example of the output?

[aaronlehmann]

I still don't think it makes sense to support v1 registries. Support for v1 push/pull is going to be removed in a few months: moby/moby#33629

[clnperez]

@stevvooe I personally did not add the image info + manifest pull code, but the image info (metadata) was being pulled to get info to populate the manifest list, IIRC. I would have to double-check, but I'll just pull the v1 bits out altogether if we're not going to support it in push/pull soon.

[aaronlehmann]

Shouldn't need the special case for windows here. In the pull code, it's so that on Linux layers can download before the image config is receive. Since this code isn't downloading layers, there's no benefit to the concurrency.

[clnperez]

Thanks!

[aaronlehmann]

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

[aaronlehmann]

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

[aaronlehmann]

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

[aaronlehmann]

Why the unmarshal/marshal instead of calling Indent on img.CanonicalJSON?

[clnperez]

I'll double-check, and maybe I was doing something else wrong, but if I didn't do the unmarshal first, it just printed ugly bytes.

[clnperez]

Nope. You're right.

[aaronlehmann]

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

[codecov-io]

Codecov Report

Merging #138 into master will increase coverage by 0.13%.
The diff coverage is 67.92%.

@@            Coverage Diff             @@
##           master     #138      +/-   ##
==========================================
+ Coverage   51.23%   51.36%   +0.13%     
==========================================
  Files         237      244       +7     
  Lines       15399    15762     +363     
==========================================
+ Hits         7889     8096     +207     
- Misses       7008     7121     +113     
- Partials      502      545      +43

[StefanScherer]

I just found out that I can use docker manifest create and docker manifest push to remote "tag" an existing image to another one.

Use case: Moving an image from one stage to another (after successful tests on different machines) without pulling and pushing the whole image. And probably able to retag Windows images as well on a Linux machine (where you can't pull and push Windows images).

PS: I tried that with manifest-tool 0.5.0, but get an error "You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion."

So +1 👍 having the "docker manifest" command.

Oh, this only works if the source manifest only has one platform, for a multiarch manifest as source the same error occurs:

$ docker manifest create stefanscherer/winspector:staging stefanscherer/winspector:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.

[stevvooe]

@StefanScherer That looks like a bug. Perhaps, if one creates a manifest list from an existing manifest list, it should clone it, rather than trying to reference it.

[stevvooe]

CPU features are being removed from OCI and we are going to deprecate them in docker's format. Let's remove them here.

[stevvooe]

@clnperez Are there a few examples of the manifest inspect output? I ask because the output of these commands tend to become what people think of when we talk about the data structure. This can be problematic, because people will look at that and say "Just add x", when that is actually impossible, because the internal data structure is misrepresented.

[estesp]

I think what @StefanScherer is looking for is more like a "manifest clone" or what might be like a docker rtag command (instead of tag locally, tag remotely) if the only difference is a tag name.

Instead, using manifest create is thinking that you want to create a manifest list of of the first entry, which is already a manifest list, hence the error message. You end up needing a new format or command to specify something more along the lines of "duplicate this same manifest in the remote registry, but with a different tag". I just added a PR for that in manifest-tool by allowing a set of "additional tags" to be provided, but a more general solution would be a remote tag operation which assumes all references and blob mounts are already handled (which would be the case for an existing manifest reference in a repo).

[clnperez]

@stevvooe here's what I had pasted in the original PR: moby/moby#27455 (comment)

Since then I changed the flag from -p for platform to -v for verbose so that it fits with both manifests and manifest lists. I extended the additional/verbose output to manifest lists so that users could also output layers (by way of the individual image's json, img.CanonicalJSON) since people seemed to be comparing layers from the previous tool's output.

I'm not aware of the the use-case, though. @StefanScherer, @estesp, @tianon, et al., do you think displaying layers of the image referenced by the manifest list is useful?

Here's an image's manifest, and it's image data (including layers). It's just the last one printed (which happens to be the arm image's info) when I run
docker-linux-amd64 manifest inspect -v clnperez/hello-world

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1462,
        "digest": "sha256:d40384c3f8619d0976e52dd6aa13e6037e16698205a95e87d5d468dc12753630"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 986,
            "digest": "sha256:a0691bf12e4effeae0f10e3e0f38e67ab16315ac870f2b35b8500ead5b5904e1"
        }
    ]
}

@stevvooe If you think having the -v is bad and will cause the scenario you described, and people think that the layers output is useful, I can remove this flag for now and we can figure out something better later after the PR is in.

[clnperez]

@StefanScherer can you paste what you were doing that Steven said sounds like a bug? I think I'm misunderstanding what you did b/c it sounds to me like you just made a manifest list with only one entry in it, which is fine. But I don't see why Phil's tool wouldn't allow that.

[stevvooe]

@clnperez Do we ever print out ImgManifestInspect? That seems useful as display. So, as I understand, we print that by default and the regular manifest with -v?

[clnperez]

@stevvooe No, we don't print that out. That's the "franken-manifest" you thought was a bad idea before. If you've reconsidered and you think it's useful I could easily put that back, and only print the manifest struct with the -v flag. Right now I print:

docker manifest inspect img-ref
manifest object

docker manifest inspect -v img-ref
manifest object
manifestdescriptor

docker manifest inspect manifest-list
DeserializedManifestList

docker manifest inspect -v manifest-list
DeserialzedManifestList
[manifest obj1, manifest obj2...]

[stevvooe]

@clnperez If you think it is useful to show the "franken manifest" and make it clear that it is not the manifest but a summary of data structures, then by all means, let's have something along those lines. Either way, this is looking good.

@dnephin What are the real blockers for getting this in as experimental, at least?

[StefanScherer]

@clnperez As Phil mentioned I'm more searching for a way to remote copy/clone a tag or manifest without pulling the image (or images which is eg. not possible for Windows images if running through a Linux Docker host).

My steps to reproduce the situation is this.

Trying to create a manifest from a manifest list which only has one platform in it works:

$ ~/Downloads/docker-darwin-amd64 manifest create stefanscherer/ngrokd:prod stefanscherer/ngrokd:staging
INFO[0000] Retrieving digests of images...              

This also can be pushed afterwards.

Trying to create a manifest from a multiarch manifest list does not work:

$ docker manifest create stefanscherer/winspector:prod stefanscherer/whoami:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.

So a "copy" or "clone" command would describe the step better than a "create" command.

[clnperez]

Thanks for the clarification @StefanScherer. That's not a bug so much as an unintended use-case. ;) I feel like we should start a list of "Features to add once this PR is in." I'm seeing a lot of cool things go into @estesp's manifest tool lately (which are exciting, b/c that means we're getting closer to multi-arch usage in the wild).

[AceHack]

Are there plans to support the yml like file format from manifest-tool?

[AceHack]

Also is there any plan to be able to update manifests one at a time? For instance, I have builds for windows and Linux and they do not have any dependencies on each other so when one finishes I just want it to go create/update it's own subtag under the main tag. Each build process can be isolated in this way.

[estesp]

@AceHack

Are there plans to support the yml like file format from manifest-tool?

yes, some features were removed from the initial PR to make it more manageable for review. As soon as @clnperez recovers from the feat of this PR, I think there are plans to create PRs to add those capabilities back in.

Also is there any plan to be able to update manifests one at a time?

A manifest list is one entity in the registry; there is no way to only modify one part of an existing entity; however, continued (re-)pushes with "sub"/architecture builds that are complete would be one way to solve this. This means your repo name/tag reference will be changing as platform entries are added/removed, which could cause downstream image user churn. As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

[AceHack]

@estesp thanks so much for your quick response. So in your 2nd comment on being able to run one at a time. I'm okay with the churn so let me ask a scenario.

If I have 2 independent build pipelines that do the following.

----Build Pipeline 1-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-1}
docker manifest annotate {multi-arch-tag} {os-specific-tag-1} --os {os-1} --arch {arch-1}
docker manifest push --purge {multi-arch-tag}

----Build Pipeline 2-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-2}
docker manifest annotate {multi-arch-tag} {os-specific-tag-2} --os {os-2} --arch {arch-2}
docker manifest push --purge {multi-arch-tag}

Is this last one wins? Where the {multi-arch-tag} has one subtag either {os-specific-tag-1} or {os-specific-tag-2}, basically the last build to run wins?
Or is it additive so that you end up with both the {os-specific-tag-1} and {os-specific-tag-2} under the main {multi-arch-tag}?

Again, thanks so much.

[tianon]

As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

Just to be clear, the root issue isn't actually solved -- still crops up from time to time because we only have a hacky workaround to help it happen less often currently. 😅 (docker-library/official-images#3835)

[clnperez]

@AceHack I have the yaml re-add on my todo for next week hopefully.

It would be nice if we could find a way to only update one part of a manifest list on a registry (maybe by pulling, copying, and repushing with original bits and updated bits).

[AceHack]

It would also be nice to use some sort of etag i.e. optimistic concurrency concept so in the unlikely case that both push at the same time there will be no concurrency funny business. One will win and the other will just get the etag failure and retry. Thanks.

[clnperez]

Here's the PR for the yaml re-add: #866

[khatribharat]

Docker CE Edge documentation for manifest command available here.

[agronholm]

I tried to create a manifest list but failed utterly. I created two images, say registry.example.org/example-amd64:staging and registry.example.org/example-arm:staging and then tried to created a manifest list using docker manifest create registry.example.org/example:staging registry.example.org/example-amd64:staging registry.example.org/example-arm:staging but got this error: no such manifest: registry.example.org/example-amd64:staging

I was under the impression that every image also has a manifest. If this is not true, how do I create the manifest for the image? This part is missing from the documentation.

[clnperez]

@agronholm would your registry happen to not be using tls? if the command can't find the manifest on a registry that meets all the default requirements, then it will just tell you one doesn't exist. if that's the case, you can use the --insecure flag on create & push.

[agronholm]

No, it is using TLS. Also, the documentation did not give me any indication that manifest create accesses the registry. I thought it was purely a local operation. That said, I did have the relevant images pushed to the registry before this. Apparently this is a requirement?

[clnperez]

Ah, yes. It sounds like there needs to be some clarification around that. It's mentioned in the insecure registries section, but if you aren't working with an insecure registry you're not going to read that section. :)

[agronholm]

Alright, so since I am not using an insecure registry, how can I figure out why it tells me there is no such manifest?

[clnperez]

I think I may have misread your previous comment:

That said, I did have the relevant images pushed to the registry before this.
Apparently this is a requirement?

The language sounded like you didn't have them pushed. Do you? That is a requirement.

[agronholm]

The language sounded like you didn't have them pushed. Do you? That is a requirement.

I did try that but it made no difference.

[clnperez]

Can you try with the -D flag and see if you get any new info?