Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add manifest command #138

Merged
merged 4 commits into from Jan 18, 2018
Merged

Add manifest command #138

merged 4 commits into from Jan 18, 2018

Conversation

@clnperez
Copy link
Contributor

@clnperez clnperez commented May 30, 2017

Enable inspection (aka "shallow pull") of images' manifest info, and also the creation of manifest lists (aka "fat manifests").

The workflow for creating a manifest list will be:

docker manifest create new-list-ref-name image-ref [image-ref...]
docker manifest annotate new-list-ref-name image-ref --os linux --arch arm
docker manifest push new-list-ref-name

There is also a manifest inspect command to allow for a "shallow pull"
of an image's manifest: docker manifest inspect manifest-or-manifest_list.

To be more in line with the existing external manifest tool, there is
also a -v option for inspect that will show information depending on
what the reference maps to (list or single manifest).

Signed-off-by: Christy Norman Perez christy@linux.vnet.ibm.com

- What I did

- How I did it

- How to verify it

- Description for the changelog

See moby/moby#27455 for history.

TODO:

  • move/refactor tests from original pr to this
@clnperez
Copy link
Contributor Author

@clnperez clnperez commented May 30, 2017

@dnephin
Copy link
Contributor

@dnephin dnephin commented May 30, 2017

Thanks for moving this PR over to the docker/cli repo!

This PR is quite large! It would be much easier to review if it could be split up and submitted incrementally. Would it be possible to include just one or two sub commands at first? The YAML config I think could be added later on.

We'd like to keep a separation between the cli/command packages (which are responsible for flag parsing, and command routing) and "client logic". Most client commands don't have much logic, but you can see this separation between the cli/compose and cli/command/stack packages. fetch.go, fetch_v1.go, and fetch_v2.go should be moved into some other package (maybe cli/manfiest).

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented May 30, 2017

@dnephin It would be a lot of work to split things back out. I realize it would be easier to review and I of course don't want to make reviewers' lives harder -- but at the same time, I would appreciate some compromise. To use your yaml suggestion as an example, I was asked to remove the yaml func, then to add it back in, so I've already been through this twice with the yaml feature alone. I'm not trying to be stubborn -- just asking for a little mercy where it can be given (without sacrificing project integrity, of course).

As for including only a few of the sub-commands, none of the ones there make sense without all the others, except maybe inspect. But, I think that's just silly to take out and put back in. 😹

As for your other point about splitting some of it out into another package, I think that makes sense and I can look into it.

@clnperez clnperez mentioned this pull request May 30, 2017
3 of 6 tasks complete

flags.StringVar(&opts.os, "os", "", "Add ios info to a manifest before pushing it.")
flags.StringVar(&opts.arch, "arch", "", "Add arch info to a manifest before pushing it.")
flags.StringSliceVar(&opts.cpuFeatures, "cpuFeatures", []string{}, "Add feature info to a manifest before pushing it.")

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

cpu-features?

This comment has been minimized.

@clnperez

clnperez Jun 1, 2017
Author Contributor

Ah, sure.

flags.StringVar(&opts.os, "os", "", "Add ios info to a manifest before pushing it.")
flags.StringVar(&opts.arch, "arch", "", "Add arch info to a manifest before pushing it.")
flags.StringSliceVar(&opts.cpuFeatures, "cpuFeatures", []string{}, "Add feature info to a manifest before pushing it.")
flags.StringSliceVar(&opts.osFeatures, "osFeatures", []string{}, "Add feature info to a manifest before pushing it.")

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

os-features?

This comment has been minimized.

@clnperez

clnperez Jun 1, 2017
Author Contributor

Yep, same. Thanks.

// Make sure the manifests are pulled, find the file you need, unmarshal the json, edit the file, and done.
targetRef, err := reference.ParseNormalizedNamed(opts.target)
if err != nil {
return fmt.Errorf("Annotate: Error parsing name for manifest list (%s): %s", opts.target, err)

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

Suggest using Wrap from github.com/pkg/errors.

service: registryService,
repoInfo: repoInfo,
}, nil
case registry.APIVersion1:

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

Manifest lists don't support v1 registries, so I'm unsure what v1ManifestFetcher is useful for. Even if v1 registries are supported in some way, I would recommend against adding support for them in a new feature.

This comment has been minimized.

@clnperez

clnperez Jun 1, 2017
Author Contributor

Maybe not clear in the initial comment, but, this subcommand isn't just for manifest lists. It also allows you to do docker manifest inspect [blah] of an image name that could be living in a v1 registry. This is so that users can do the "shallow pull" of an image that several people had requested in earlier issues

This comment has been minimized.

@stevvooe

stevvooe Jun 14, 2017
Contributor

@clnperez How do v1 registries have manifests? Do you have an example of the output?

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 16, 2017
Contributor

I still don't think it makes sense to support v1 registries. Support for v1 push/pull is going to be removed in a few months: moby/moby#33629

This comment has been minimized.

@clnperez

clnperez Jun 19, 2017
Author Contributor

@stevvooe I personally did not add the image info + manifest pull code, but the image info (metadata) was being pulled to get info to populate the manifest list, IIRC. I would have to double-check, but I'll just pull the v1 bits out altogether if we're not going to support it in push/pull soon.

configJSON []byte // raw serialized image config
unmarshalledConfig image.Image // deserialized image config
)
if runtime.GOOS == "windows" {

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

Shouldn't need the special case for windows here. In the pull code, it's so that on Linux layers can download before the image config is receive. Since this code isn't downloading layers, there's no benefit to the concurrency.

This comment has been minimized.

@clnperez

clnperez Jun 1, 2017
Author Contributor

Thanks!

if err != nil {
return err
}
fmt.Fprintf(dockerCli.Out(), "%s\n", prettyJSON.String())

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

if err != nil {
return err
}
fmt.Fprintf(dockerCli.Out(), "%s\n", prettyJSON.String())

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

if err != nil {
return err
}
fmt.Fprintf(dockerCli.Out(), "%s\n", prettyJSON.String())

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

return err
}
prettyJSON.Reset()
err = json.Indent(&prettyJSON, jsonBytes, "", " ")

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

Why the unmarshal/marshal instead of calling Indent on img.CanonicalJSON?

This comment has been minimized.

@clnperez

clnperez Jun 1, 2017
Author Contributor

I'll double-check, and maybe I was doing something else wrong, but if I didn't do the unmarshal first, it just printed ugly bytes.

This comment has been minimized.

@clnperez

clnperez Jun 2, 2017
Author Contributor

Nope. You're right.

if err != nil {
return err
}
fmt.Fprintf(dockerCli.Out(), "%s\n", prettyJSON.String())

This comment has been minimized.

@aaronlehmann

aaronlehmann Jun 1, 2017
Contributor

fmt.Fprintln(dockerCli.Out(), prettyJSON.String())

@clnperez clnperez force-pushed the clnperez:manifest-cmd branch from 42b0542 to 038fdf4 Jun 2, 2017
@codecov-io
Copy link

@codecov-io codecov-io commented Jun 2, 2017

Codecov Report

Merging #138 into master will increase coverage by 0.13%.
The diff coverage is 67.92%.

@@            Coverage Diff             @@
##           master     #138      +/-   ##
==========================================
+ Coverage   51.23%   51.36%   +0.13%     
==========================================
  Files         237      244       +7     
  Lines       15399    15762     +363     
==========================================
+ Hits         7889     8096     +207     
- Misses       7008     7121     +113     
- Partials      502      545      +43
@clnperez clnperez force-pushed the clnperez:manifest-cmd branch 2 times, most recently from 063558f to dc0fc04 Jun 2, 2017
@StefanScherer
Copy link
Member

@StefanScherer StefanScherer commented Jun 14, 2017

I just found out that I can use docker manifest create and docker manifest push to remote "tag" an existing image to another one.

Use case: Moving an image from one stage to another (after successful tests on different machines) without pulling and pushing the whole image. And probably able to retag Windows images as well on a Linux machine (where you can't pull and push Windows images).

PS: I tried that with manifest-tool 0.5.0, but get an error "You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion."

So +1 👍 having the "docker manifest" command.

Oh, this only works if the source manifest only has one platform, for a multiarch manifest as source the same error occurs:

$ docker manifest create stefanscherer/winspector:staging stefanscherer/winspector:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.
@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Jun 14, 2017

@StefanScherer That looks like a bug. Perhaps, if one creates a manifest list from an existing manifest list, it should clone it, rather than trying to reference it.

variant string // an architecture variant
os string
arch string
cpuFeatures []string

This comment has been minimized.

@stevvooe

stevvooe Jun 14, 2017
Contributor

CPU features are being removed from OCI and we are going to deprecate them in docker's format. Let's remove them here.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Jun 14, 2017

@clnperez Are there a few examples of the manifest inspect output? I ask because the output of these commands tend to become what people think of when we talk about the data structure. This can be problematic, because people will look at that and say "Just add x", when that is actually impossible, because the internal data structure is misrepresented.

@estesp
Copy link
Contributor

@estesp estesp commented Jun 14, 2017

I think what @StefanScherer is looking for is more like a "manifest clone" or what might be like a docker rtag command (instead of tag locally, tag remotely) if the only difference is a tag name.

Instead, using manifest create is thinking that you want to create a manifest list of of the first entry, which is already a manifest list, hence the error message. You end up needing a new format or command to specify something more along the lines of "duplicate this same manifest in the remote registry, but with a different tag". I just added a PR for that in manifest-tool by allowing a set of "additional tags" to be provided, but a more general solution would be a remote tag operation which assumes all references and blob mounts are already handled (which would be the case for an existing manifest reference in a repo).

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Jun 14, 2017

@stevvooe here's what I had pasted in the original PR: moby/moby#27455 (comment)

Since then I changed the flag from -p for platform to -v for verbose so that it fits with both manifests and manifest lists. I extended the additional/verbose output to manifest lists so that users could also output layers (by way of the individual image's json, img.CanonicalJSON) since people seemed to be comparing layers from the previous tool's output.

I'm not aware of the the use-case, though. @StefanScherer, @estesp, @tianon, et al., do you think displaying layers of the image referenced by the manifest list is useful?

Here's an image's manifest, and it's image data (including layers). It's just the last one printed (which happens to be the arm image's info) when I run
docker-linux-amd64 manifest inspect -v clnperez/hello-world

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1462,
        "digest": "sha256:d40384c3f8619d0976e52dd6aa13e6037e16698205a95e87d5d468dc12753630"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 986,
            "digest": "sha256:a0691bf12e4effeae0f10e3e0f38e67ab16315ac870f2b35b8500ead5b5904e1"
        }
    ]
}

@stevvooe If you think having the -v is bad and will cause the scenario you described, and people think that the layers output is useful, I can remove this flag for now and we can figure out something better later after the PR is in.

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Jun 14, 2017

@StefanScherer can you paste what you were doing that Steven said sounds like a bug? I think I'm misunderstanding what you did b/c it sounds to me like you just made a manifest list with only one entry in it, which is fine. But I don't see why Phil's tool wouldn't allow that.

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Jun 14, 2017

@clnperez Do we ever print out ImgManifestInspect? That seems useful as display. So, as I understand, we print that by default and the regular manifest with -v?

@clnperez clnperez force-pushed the clnperez:manifest-cmd branch from dc0fc04 to 81f1bb3 Jun 15, 2017
@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Jun 15, 2017

@stevvooe No, we don't print that out. That's the "franken-manifest" you thought was a bad idea before. If you've reconsidered and you think it's useful I could easily put that back, and only print the manifest struct with the -v flag. Right now I print:

docker manifest inspect img-ref
manifest object

docker manifest inspect -v img-ref
manifest object
manifestdescriptor

docker manifest inspect manifest-list
DeserializedManifestList

docker manifest inspect -v manifest-list
DeserialzedManifestList
[manifest obj1, manifest obj2...]

@stevvooe
Copy link
Contributor

@stevvooe stevvooe commented Jun 15, 2017

@clnperez If you think it is useful to show the "franken manifest" and make it clear that it is not the manifest but a summary of data structures, then by all means, let's have something along those lines. Either way, this is looking good.

@dnephin What are the real blockers for getting this in as experimental, at least?

@StefanScherer
Copy link
Member

@StefanScherer StefanScherer commented Jun 15, 2017

@clnperez As Phil mentioned I'm more searching for a way to remote copy/clone a tag or manifest without pulling the image (or images which is eg. not possible for Windows images if running through a Linux Docker host).

My steps to reproduce the situation is this.

Trying to create a manifest from a manifest list which only has one platform in it works:

$ ~/Downloads/docker-darwin-amd64 manifest create stefanscherer/ngrokd:prod stefanscherer/ngrokd:staging
INFO[0000] Retrieving digests of images...              

This also can be pushed afterwards.

Trying to create a manifest from a multiarch manifest list does not work:

$ docker manifest create stefanscherer/winspector:prod stefanscherer/whoami:latest
INFO[0000] Retrieving digests of images...              
You specified a manifest list entry from a digest that points to a current manifest list. Manifest lists do not allow recursion.

So a "copy" or "clone" command would describe the step better than a "create" command.

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Jun 15, 2017

Thanks for the clarification @StefanScherer. That's not a bug so much as an unintended use-case. ;) I feel like we should start a list of "Features to add once this PR is in." I'm seeing a lot of cool things go into @estesp's manifest tool lately (which are exciting, b/c that means we're getting closer to multi-arch usage in the wild).

@clnperez clnperez force-pushed the clnperez:manifest-cmd branch from 81f1bb3 to e34188d Jun 15, 2017
@AceHack
Copy link

@AceHack AceHack commented Jan 31, 2018

Are there plans to support the yml like file format from manifest-tool?

@AceHack
Copy link

@AceHack AceHack commented Jan 31, 2018

Also is there any plan to be able to update manifests one at a time? For instance, I have builds for windows and Linux and they do not have any dependencies on each other so when one finishes I just want it to go create/update it's own subtag under the main tag. Each build process can be isolated in this way.

@estesp
Copy link
Contributor

@estesp estesp commented Jan 31, 2018

@AceHack

Are there plans to support the yml like file format from manifest-tool?

yes, some features were removed from the initial PR to make it more manageable for review. As soon as @clnperez recovers from the feat of this PR, I think there are plans to create PRs to add those capabilities back in.

Also is there any plan to be able to update manifests one at a time?

A manifest list is one entity in the registry; there is no way to only modify one part of an existing entity; however, continued (re-)pushes with "sub"/architecture builds that are complete would be one way to solve this. This means your repo name/tag reference will be changing as platform entries are added/removed, which could cause downstream image user churn. As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

@AceHack
Copy link

@AceHack AceHack commented Jan 31, 2018

@estesp thanks so much for your quick response. So in your 2nd comment on being able to run one at a time. I'm okay with the churn so let me ask a scenario.

If I have 2 independent build pipelines that do the following.

----Build Pipeline 1-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-1}
docker manifest annotate {multi-arch-tag} {os-specific-tag-1} --os {os-1} --arch {arch-1}
docker manifest push --purge {multi-arch-tag}

----Build Pipeline 2-----
docker manifest create --amend {multi-arch-tag} {os-specific-tag-2}
docker manifest annotate {multi-arch-tag} {os-specific-tag-2} --os {os-2} --arch {arch-2}
docker manifest push --purge {multi-arch-tag}

Is this last one wins? Where the {multi-arch-tag} has one subtag either {os-specific-tag-1} or {os-specific-tag-2}, basically the last build to run wins?
Or is it additive so that you end up with both the {os-specific-tag-1} and {os-specific-tag-2} under the main {multi-arch-tag}?

Again, thanks so much.

@tianon
Copy link
Contributor

@tianon tianon commented Jan 31, 2018

As an example, there was a similar issue last year with a popular official image that was pushed first by an ARM/ARM64 build completion without amd64 being done.. causing consternation that that main official image was not operational for several hours on amd64.

Just to be clear, the root issue isn't actually solved -- still crops up from time to time because we only have a hacky workaround to help it happen less often currently. 😅 (docker-library/official-images#3835)

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Jan 31, 2018

@AceHack I have the yaml re-add on my todo for next week hopefully.

It would be nice if we could find a way to only update one part of a manifest list on a registry (maybe by pulling, copying, and repushing with original bits and updated bits).

@AceHack
Copy link

@AceHack AceHack commented Feb 1, 2018

It would also be nice to use some sort of etag i.e. optimistic concurrency concept so in the unlikely case that both push at the same time there will be no concurrency funny business. One will win and the other will just get the etag failure and retry. Thanks.

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Feb 8, 2018

Here's the PR for the yaml re-add: #866

@khatribharat
Copy link

@khatribharat khatribharat commented Feb 17, 2018

Docker CE Edge documentation for manifest command available here.

@agronholm
Copy link

@agronholm agronholm commented Feb 18, 2018

I tried to create a manifest list but failed utterly. I created two images, say registry.example.org/example-amd64:staging and registry.example.org/example-arm:staging and then tried to created a manifest list using docker manifest create registry.example.org/example:staging registry.example.org/example-amd64:staging registry.example.org/example-arm:staging but got this error: no such manifest: registry.example.org/example-amd64:staging

I was under the impression that every image also has a manifest. If this is not true, how do I create the manifest for the image? This part is missing from the documentation.

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Feb 18, 2018

@agronholm would your registry happen to not be using tls? if the command can't find the manifest on a registry that meets all the default requirements, then it will just tell you one doesn't exist. if that's the case, you can use the --insecure flag on create & push.

@agronholm
Copy link

@agronholm agronholm commented Feb 18, 2018

No, it is using TLS. Also, the documentation did not give me any indication that manifest create accesses the registry. I thought it was purely a local operation. That said, I did have the relevant images pushed to the registry before this. Apparently this is a requirement?

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Feb 19, 2018

Ah, yes. It sounds like there needs to be some clarification around that. It's mentioned in the insecure registries section, but if you aren't working with an insecure registry you're not going to read that section. :)

@agronholm
Copy link

@agronholm agronholm commented Feb 19, 2018

Alright, so since I am not using an insecure registry, how can I figure out why it tells me there is no such manifest?

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Feb 19, 2018

I think I may have misread your previous comment:

That said, I did have the relevant images pushed to the registry before this.
Apparently this is a requirement?

The language sounded like you didn't have them pushed. Do you? That is a requirement.

@agronholm
Copy link

@agronholm agronholm commented Feb 20, 2018

The language sounded like you didn't have them pushed. Do you? That is a requirement.

I did try that but it made no difference.

@clnperez
Copy link
Contributor Author

@clnperez clnperez commented Feb 20, 2018

Can you try with the -D flag and see if you get any new info?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.