New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add manifest command to docker cli #27455

Closed
wants to merge 1 commit into
from

Conversation

@clnperez
Contributor

clnperez commented Oct 17, 2016

- What I did
Porting the manifest tool code from https://github.com/estesp/manifest-tool

Version One workflow:

  • manifest inspect:
    docker manifest inspect busybox:latest

  • manifest create list:

  1. Create a yaml file, e.g.
image: docker.io/clnperez/hello-world:latest
manifests:
  -
    image: docker.io/ppc64le/hello-world:latest
    platform:
      architecture: ppc64le
      os: linux
  -
    image: docker.io/hello-world:latest
    platform:
      architecture: amd64
      features:
        - sse
      variant: v_a
      os: linux

  1. docker manifest create busybox-manifest.yaml

Version Two Three Workflow:

  • inspect: will pull down and print out a manifest or manifest list
  • annotate: will modify the locally-stored manifest list by adding
    variants, os features, cpu features, an os kind, and/or an archicture

The inspect, fetch, and annotate commands are optional if all a user
wants to do is create a multi-arch manifest list based on existing
images.

Things left to do:

  • Add support for plain http registries.
  • Probably move the manifest storage out from ~/.docker/manifests (and
    figure out how to handle multiple users possibly doing simultaneous
    annotations of the same image manifest).
  • Lots of cleanup.
  • Lots more tests (and make existing ones much more thorough).
  • Fix a the user lookup issue when running a static binary (probably fixed by moving manifests outside a user dir).
  • Move borrowed/duplicated code from moby/moby/pull/29478 into a sharable place (in a separate PR)

- How to verify it

  • make binary

  • Create a new unaltered fat manifest:
    ./bundles/latest/dynbinary-client/docker manifest create --name reponame/busybox busybox ppc64le/busybox

  • Annotate a manifest inside of the new list context:
    ./bundles/latest/dynbinary-client/docker manifest annotate reponame/busybox ppc64le/busybox --os linux --arch ppc64le
    ./bundles/latest/dynbinary-client/docker manifest annotate reponame/busybox busybox --osFeatures osf1,osf2

  • Push the manifest list to a registry:
    ./bundles/latest/dynbinary-client/docker manifest push reponame/busybox

  • Display [committed] manifest or manifest list details:
    ./bundles/latest/dynbinary-client/docker manifest inspect ppc64le/busybox
    ./bundles/latest/dynbinary-client/docker manifest inspect reponame/busybox

  • Pull (to verify)
    docker run --rm reponame/busybox uname -m
    Will pull the correct image down for your platform, and print your system's hw name.
    - Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

manatee pulling baby manatee

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Oct 17, 2016

Contributor

@tophj-ibm is working on tests as well.

Contributor

clnperez commented Oct 17, 2016

@tophj-ibm is working on tests as well.

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom Oct 17, 2016

Member

is this just a new command for building manifest lists or?

Member

runcom commented Oct 17, 2016

is this just a new command for building manifest lists or?

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Oct 17, 2016

Contributor

@runcom building and also fetching/displaying them. When we'd talked to @RichardScothern @dmcgowan several weeks ago they said a good first step would be to just get a PoC fetch working, so that's what this is.

Contributor

clnperez commented Oct 17, 2016

@runcom building and also fetching/displaying them. When we'd talked to @RichardScothern @dmcgowan several weeks ago they said a good first step would be to just get a PoC fetch working, so that's what this is.

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Oct 17, 2016

Contributor

@runcom both :) first step of a more comprehensive manifest subcommand in response to this comment from @stevvooe : #24739 (comment)

Contributor

estesp commented Oct 17, 2016

@runcom both :) first step of a more comprehensive manifest subcommand in response to this comment from @stevvooe : #24739 (comment)

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom Oct 17, 2016

Member

@runcom both :) first step of a more comprehensive manifest subcommand in response to this comment from @stevvooe : #24739 (comment)

figured that thx! we've been trying the same in docker/distribution#1252 also, feel free to add a Close to close that proposal as well..

Member

runcom commented Oct 17, 2016

@runcom both :) first step of a more comprehensive manifest subcommand in response to this comment from @stevvooe : #24739 (comment)

figured that thx! we've been trying the same in docker/distribution#1252 also, feel free to add a Close to close that proposal as well..

Show outdated Hide outdated cli/command/manifest/util.go
// Added linux/s390x as we know System z support already exists
var validOSArch = map[string]bool{
"darwin/386": true,

This comment has been minimized.

@stevvooe

stevvooe Oct 17, 2016

Contributor

Where is this microformat defined?

@stevvooe

stevvooe Oct 17, 2016

Contributor

Where is this microformat defined?

This comment has been minimized.

@estesp

estesp Oct 17, 2016

Contributor

Hey @stevvooe ! Anything ugly in this PR is probably a copy over from my external "manifest-tool" project :) In this case, I was trying to validate the input against a known universe of Golang os+arch pairs just to sanitize what people might claim as valid pairs. The format itself is just a slash-separated copy of the Golang list of accepted pairs from golang.org. Of course this in itself is also broad because the Docker engine does not exist for every pair in that list.

@estesp

estesp Oct 17, 2016

Contributor

Hey @stevvooe ! Anything ugly in this PR is probably a copy over from my external "manifest-tool" project :) In this case, I was trying to validate the input against a known universe of Golang os+arch pairs just to sanitize what people might claim as valid pairs. The format itself is just a slash-separated copy of the Golang list of accepted pairs from golang.org. Of course this in itself is also broad because the Docker engine does not exist for every pair in that list.

Show outdated Hide outdated cli/command/manifest/util.go
func isValidOSArch(os string, arch string) bool {
osarch := fmt.Sprintf("%s/%s", os, arch)
_, ok := validOSArch[osarch]

This comment has been minimized.

@stevvooe

stevvooe Oct 17, 2016

Contributor

You can define the map with a struct key to avoid having to format a string here.

@stevvooe

stevvooe Oct 17, 2016

Contributor

You can define the map with a struct key to avoid having to format a string here.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Oct 17, 2016

Contributor

@clnperez Thanks for the contribution! It will be great to get this going!

@estesp While the demo is informative, do we have a way of doing this without introducing a yaml file format? Note that #24739 contains a lot of "modern" requirements.

It would be good to get a clear design of what this tool actually does. I think a short list would be:

  • inspect current manifests
  • shallow pull remote manifests
  • take manifests that are partially pulled into full images
  • assemble multi-arch manifests for images or other sub-manifests

To accomplish this, we need to define what a manifest is as an object in docker. Up to this point, it is largely hidden from the user.

@runcom I am not sure if this fully covers docker/distribution#1252. Inspecting the registry contents and the push/pull artifacts on the engine-side may be fundamentally different operations in practice.

Contributor

stevvooe commented Oct 17, 2016

@clnperez Thanks for the contribution! It will be great to get this going!

@estesp While the demo is informative, do we have a way of doing this without introducing a yaml file format? Note that #24739 contains a lot of "modern" requirements.

It would be good to get a clear design of what this tool actually does. I think a short list would be:

  • inspect current manifests
  • shallow pull remote manifests
  • take manifests that are partially pulled into full images
  • assemble multi-arch manifests for images or other sub-manifests

To accomplish this, we need to define what a manifest is as an object in docker. Up to this point, it is largely hidden from the user.

@runcom I am not sure if this fully covers docker/distribution#1252. Inspecting the registry contents and the push/pull artifacts on the engine-side may be fundamentally different operations in practice.

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom Oct 17, 2016

Member

@stevvooe eventually I'm sure it will

Member

runcom commented Oct 17, 2016

@stevvooe eventually I'm sure it will

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Oct 17, 2016

Contributor

@stevvooe The idea of not using the yaml file occurred to me as well, so if we can find a way not to pull in that dependency to the docker codebase for just that I think it would be nice. But I personally couldn't think of a simpler way (maybe json instead but it's just not as user-friendly).

I updated the comment with the current short list so hopefully that's helpful. I agree that we need to figure out a way to frame what this all means for a docker user. IIUC, this PR has come about for two [main] reasons:

  1. Users wanting to be able to do something like a 'shallow pull' or an image inspect but without pulling the image. So we can fetch the manifest and display it.
  2. The need to stop tagging arch-specific images as arch-specific (the assemble), and also to be able to find out which images support your architecture (the fetch).

So keeping those in mind when defining the concept to users might be beneficial. (Have I missed one?)

A point of confusion I can see for the "shallow pull" case is if a user gets back a list but was only expecting a single group of size, digest, platform information. Maybe a default would be only to display the manifest that would run on their platform, and add a flag for --list, which would show all the images.

A question I have is whether we need to store this. I would say no (though not in an authoritative way by any means).

Contributor

clnperez commented Oct 17, 2016

@stevvooe The idea of not using the yaml file occurred to me as well, so if we can find a way not to pull in that dependency to the docker codebase for just that I think it would be nice. But I personally couldn't think of a simpler way (maybe json instead but it's just not as user-friendly).

I updated the comment with the current short list so hopefully that's helpful. I agree that we need to figure out a way to frame what this all means for a docker user. IIUC, this PR has come about for two [main] reasons:

  1. Users wanting to be able to do something like a 'shallow pull' or an image inspect but without pulling the image. So we can fetch the manifest and display it.
  2. The need to stop tagging arch-specific images as arch-specific (the assemble), and also to be able to find out which images support your architecture (the fetch).

So keeping those in mind when defining the concept to users might be beneficial. (Have I missed one?)

A point of confusion I can see for the "shallow pull" case is if a user gets back a list but was only expecting a single group of size, digest, platform information. Maybe a default would be only to display the manifest that would run on their platform, and add a flag for --list, which would show all the images.

A question I have is whether we need to store this. I would say no (though not in an authoritative way by any means).

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Oct 18, 2016

Contributor

@clnperez An completely contrived example of a yaml-less approach would be something like this:

docker manifest pull mybuildregistry:5000/redis-arm 
docker manifest pull mybuildregistry:5000/redis-s390x
docker manifest pull mybuildregistry:5000/redis-x86
docker manifest annotate --platform.features sse4 mybuildregistry:5000/redis-x86
docker manifest assemble mybuildregistry:5000/redis mybuildregistry:5000/redis-arm mybuildregistry:5000/redis-s390x mybuildregistry:5000/redis-x86
docker manifest push mybuildregistry:5000/redis
Contributor

stevvooe commented Oct 18, 2016

@clnperez An completely contrived example of a yaml-less approach would be something like this:

docker manifest pull mybuildregistry:5000/redis-arm 
docker manifest pull mybuildregistry:5000/redis-s390x
docker manifest pull mybuildregistry:5000/redis-x86
docker manifest annotate --platform.features sse4 mybuildregistry:5000/redis-x86
docker manifest assemble mybuildregistry:5000/redis mybuildregistry:5000/redis-arm mybuildregistry:5000/redis-s390x mybuildregistry:5000/redis-x86
docker manifest push mybuildregistry:5000/redis
@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Oct 18, 2016

Contributor

@stevvooe agree that the YAML parsing/input adds a lot of potentially unnecessary code to the PR, and agree that we can probably find a "YAML-less" approach that works (similar to the flow in your comment). It made for a simple demonstration of the assembly steps, so was reasonable for my PoC. We do need to figure out the input method for the features and variant information that the platform object is capable to hold, which will not be part of the image layer/manifest content, at least not at the moment.

Contributor

estesp commented Oct 18, 2016

@stevvooe agree that the YAML parsing/input adds a lot of potentially unnecessary code to the PR, and agree that we can probably find a "YAML-less" approach that works (similar to the flow in your comment). It made for a simple demonstration of the assembly steps, so was reasonable for my PoC. We do need to figure out the input method for the features and variant information that the platform object is capable to hold, which will not be part of the image layer/manifest content, at least not at the moment.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Oct 18, 2016

Contributor

@estesp I agree.

We do need to figure out the input method for the features and variant information that the platform object is capable to hold, which will not be part of the image layer/manifest content, at least not at the moment.

Not sure if you saw this line in my example:

docker manifest annotate --platform.features sse4 mybuildregistry:5000/redis-x86
Contributor

stevvooe commented Oct 18, 2016

@estesp I agree.

We do need to figure out the input method for the features and variant information that the platform object is capable to hold, which will not be part of the image layer/manifest content, at least not at the moment.

Not sure if you saw this line in my example:

docker manifest annotate --platform.features sse4 mybuildregistry:5000/redis-x86
@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp Oct 18, 2016

Contributor

Doh. Speed reading getting me in trouble again :) that seems fine

Sent from my iPhone

On Oct 18, 2016, at 7:02 PM, Stephen Day notifications@github.com wrote:

@estesp I agree.

We do need to figure out the input method for the features and variant information that the platform object is capable to hold, which will not be part of the image layer/manifest content, at least not at the moment.

Not sure if you saw this line in my example:

docker manifest annotate --platform.features sse4 mybuildregistry:5000/redis-x86

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Contributor

estesp commented Oct 18, 2016

Doh. Speed reading getting me in trouble again :) that seems fine

Sent from my iPhone

On Oct 18, 2016, at 7:02 PM, Stephen Day notifications@github.com wrote:

@estesp I agree.

We do need to figure out the input method for the features and variant information that the platform object is capable to hold, which will not be part of the image layer/manifest content, at least not at the moment.

Not sure if you saw this line in my example:

docker manifest annotate --platform.features sse4 mybuildregistry:5000/redis-x86

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Oct 19, 2016

Contributor

@stevvooe I think it's good to be good to be able to create the list without first having to pull down the manifests, too.

Contributor

clnperez commented Oct 19, 2016

@stevvooe I think it's good to be good to be able to create the list without first having to pull down the manifests, too.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Oct 19, 2016

Contributor

@clnperez

@stevvooe I think it's good to be good to be able to create the list without first having to pull down the manifests, too.

How would this work? Where do you get the manifests from initially?

Contributor

stevvooe commented Oct 19, 2016

@clnperez

@stevvooe I think it's good to be good to be able to create the list without first having to pull down the manifests, too.

How would this work? Where do you get the manifests from initially?

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Oct 19, 2016

Contributor

@stevvooe The way this PR (and @estesp's tool) works currently is, using what's in the yaml file, it does all the pulls for the user under the covers (which is what I meant, in case you thought I meant not to ever pull them down at all):

> docker manifest create hello-world-manifest.yaml                                                                                
INFO[0000] Retrieving digests of images...              
INFO[0003] Image "docker.io/ppc64le/hello-world:latest" is digest sha256:4145a08320bcfbc55c4005ddae4e4048af16f3abc754c721801ab88b53710033; size: 503 
INFO[0006] Image "docker.io/hello-world:latest" is digest sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9; size: 502
Digest: sha256:0fb0350d85ea3cef108f5494b95314d7df66e2d89ba23c734ebd27d2a64fe677

Edit: Adding the Digest line that didn't get pasted.

One thing that Phil had mentioned as a todo that would be nice would be to do some introspection and create the fat manifest based on the architectures stored in the manifests that step pulls.

So if you start with the yaml that I used for the above example:

image: docker.io/clnperez/hello-world:latest
manifests:
  -
    image: docker.io/ppc64le/hello-world:latest
    platform:
      architecture: ppc64le
      os: linux
  -
    image: docker.io/hello-world:latest
    platform:
      architecture: amd64
      features:
        - sse
      os: linux

You could cut out the platform bits, and maybe move all that into a single command. There could be a "dry-run" flag, too, so you can dump what the list would look like. Then the entire flow could go something like:

docker login
docker manifest create [--dry-run] digest1 [-f d1feature1,d1feature2...] [digest2[-f...]|digest3[-f...]|...]

So the same example as above would be:

docker manifest create docker.io/clnperez/hello-world:latest  ppc64le/hello-world:latest hello-world:latest -f sse

The good thing about having the file (yaml or otherwise) is that if you want to just change one digest in the manifest list you can source-control it, etc., the commands are a lot shorter, and you don't have to re-type the whole command every time. But maybe you can do that with a file of commands anyway. :)

@stevvooe WDYT about this command flow?

Contributor

clnperez commented Oct 19, 2016

@stevvooe The way this PR (and @estesp's tool) works currently is, using what's in the yaml file, it does all the pulls for the user under the covers (which is what I meant, in case you thought I meant not to ever pull them down at all):

> docker manifest create hello-world-manifest.yaml                                                                                
INFO[0000] Retrieving digests of images...              
INFO[0003] Image "docker.io/ppc64le/hello-world:latest" is digest sha256:4145a08320bcfbc55c4005ddae4e4048af16f3abc754c721801ab88b53710033; size: 503 
INFO[0006] Image "docker.io/hello-world:latest" is digest sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9; size: 502
Digest: sha256:0fb0350d85ea3cef108f5494b95314d7df66e2d89ba23c734ebd27d2a64fe677

Edit: Adding the Digest line that didn't get pasted.

One thing that Phil had mentioned as a todo that would be nice would be to do some introspection and create the fat manifest based on the architectures stored in the manifests that step pulls.

So if you start with the yaml that I used for the above example:

image: docker.io/clnperez/hello-world:latest
manifests:
  -
    image: docker.io/ppc64le/hello-world:latest
    platform:
      architecture: ppc64le
      os: linux
  -
    image: docker.io/hello-world:latest
    platform:
      architecture: amd64
      features:
        - sse
      os: linux

You could cut out the platform bits, and maybe move all that into a single command. There could be a "dry-run" flag, too, so you can dump what the list would look like. Then the entire flow could go something like:

docker login
docker manifest create [--dry-run] digest1 [-f d1feature1,d1feature2...] [digest2[-f...]|digest3[-f...]|...]

So the same example as above would be:

docker manifest create docker.io/clnperez/hello-world:latest  ppc64le/hello-world:latest hello-world:latest -f sse

The good thing about having the file (yaml or otherwise) is that if you want to just change one digest in the manifest list you can source-control it, etc., the commands are a lot shorter, and you don't have to re-type the whole command every time. But maybe you can do that with a file of commands anyway. :)

@stevvooe WDYT about this command flow?

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Oct 19, 2016

Contributor

@clnperez docker manifest assemble mybuildregistry:5000/redis mybuildregistry:5000/redis-arm mybuildregistry:5000/redis-s390x mybuildregistry:5000/redis-x86 d could always do the pull.

YAML (or any other file format) isn't needed here at all. This kind of information should be part of the upstream's image config.

I'd recommend avoiding trying to intersperse flags that apply to arg values. It is fragile and hard to use. I'm not sure the annotate command is quite right, but if we could pull out a handle to manipulate, that would allow the user to insert the right data. Really, what is happening here, is several descriptors are being staged for assembly. One needs to be able to create a set of descriptors based on the upstream content, annotate the descriptors, then assemble that set of descriptors.

Contributor

stevvooe commented Oct 19, 2016

@clnperez docker manifest assemble mybuildregistry:5000/redis mybuildregistry:5000/redis-arm mybuildregistry:5000/redis-s390x mybuildregistry:5000/redis-x86 d could always do the pull.

YAML (or any other file format) isn't needed here at all. This kind of information should be part of the upstream's image config.

I'd recommend avoiding trying to intersperse flags that apply to arg values. It is fragile and hard to use. I'm not sure the annotate command is quite right, but if we could pull out a handle to manipulate, that would allow the user to insert the right data. Really, what is happening here, is several descriptors are being staged for assembly. One needs to be able to create a set of descriptors based on the upstream content, annotate the descriptors, then assemble that set of descriptors.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Oct 20, 2016

Contributor

@stevvooe Yah, I didn't much like the interspersed flags there either. Just trying to find a way to cut this down into fewer commands.

As for the file, I understand it's not needed, but was just pointing out how it could be useful for users who are going to be pushing these often.

Contributor

clnperez commented Oct 20, 2016

@stevvooe Yah, I didn't much like the interspersed flags there either. Just trying to find a way to cut this down into fewer commands.

As for the file, I understand it's not needed, but was just pointing out how it could be useful for users who are going to be pushing these often.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Dec 1, 2016

Contributor

@stevvooe It took a month, but it's way different now, and and uses the commands you suggested. It's far from perfect, but how does this sound? (I updated the description to reflect the new commands.)

Contributor

clnperez commented Dec 1, 2016

@stevvooe It took a month, but it's way different now, and and uses the commands you suggested. It's far from perfect, but how does this sound? (I updated the description to reflect the new commands.)

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Dec 1, 2016

Contributor

My biggest issue with the way this works now (under the covers) is that it does a fetch of a manifest for everything (every inspect, every fetch, annotate, and create) even though the manifests are locally stored (named as their digest hash). I'm sure there's a way around that but that's how it works at present.

Contributor

clnperez commented Dec 1, 2016

My biggest issue with the way this works now (under the covers) is that it does a fetch of a manifest for everything (every inspect, every fetch, annotate, and create) even though the manifests are locally stored (named as their digest hash). I'm sure there's a way around that but that's how it works at present.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Dec 5, 2016

Contributor

Pinging a few other people I didn't before for general input. @duglin @tianon

Feel free to pull in anyone else for usage +1's or -1s. 😃

Contributor

clnperez commented Dec 5, 2016

Pinging a few other people I didn't before for general input. @duglin @tianon

Feel free to pull in anyone else for usage +1's or -1s. 😃

@tianon

This comment has been minimized.

Show comment
Hide comment
@tianon

tianon Dec 5, 2016

Member

In discussing this sort of functionality with users, I've heard many express interest in the ability to construct a manifest list somewhat ad-hoc from a group of machines. I think this comment from @StefanScherer sums up the expectations around multi-arch/manifest list support I've seen while talking to folks.

Essentially, a way for a given build machine to build an image for the current architecture and "push" that image into the appropriate architecture slot in an existing manifest list (or create a new one) in a reasonably safe way would cover a lot of the most simple use cases right out of the box. For example, if we're building golang:latest for both Linux on amd64 and Nano Server, then having both architectures build/push elsewhere, then a third process which builds the manifest list is a bit hairy, and it'd be neat if it were possible to instead simply push directly to golang:latest from both places (with a new subcommand or flag) and have it add to an existing manifest.

It looks like something like this could be implemented by using inspect, etc, but that does still leave the problem of needing to push an image explicitly before it can be added to a manifest list, right? (hoping I've missed something 😇)

Member

tianon commented Dec 5, 2016

In discussing this sort of functionality with users, I've heard many express interest in the ability to construct a manifest list somewhat ad-hoc from a group of machines. I think this comment from @StefanScherer sums up the expectations around multi-arch/manifest list support I've seen while talking to folks.

Essentially, a way for a given build machine to build an image for the current architecture and "push" that image into the appropriate architecture slot in an existing manifest list (or create a new one) in a reasonably safe way would cover a lot of the most simple use cases right out of the box. For example, if we're building golang:latest for both Linux on amd64 and Nano Server, then having both architectures build/push elsewhere, then a third process which builds the manifest list is a bit hairy, and it'd be neat if it were possible to instead simply push directly to golang:latest from both places (with a new subcommand or flag) and have it add to an existing manifest.

It looks like something like this could be implemented by using inspect, etc, but that does still leave the problem of needing to push an image explicitly before it can be added to a manifest list, right? (hoping I've missed something 😇)

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Dec 5, 2016

Contributor

@clnperez Thanks for taking the feedback into account! Sorry for not being more responsive.

Do you have a doc with a API flow example in it? To be honest, it is hard to just read the code and infer if it is right or not.

It might be a good idea to get agreement there and then review against that.

Contributor

stevvooe commented Dec 5, 2016

@clnperez Thanks for taking the feedback into account! Sorry for not being more responsive.

Do you have a doc with a API flow example in it? To be honest, it is hard to just read the code and infer if it is right or not.

It might be a good idea to get agreement there and then review against that.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Dec 6, 2016

Contributor

@stevvooe I updated the initial comment with a couple of flow examples. Is that helpful?

Honestly at this point I'd rather have a little more discussion around that flow before really having people digging into the code. Things there still need some work. I just wanted a working prototype for people to play with for now.

@tianon and @StefanScherer -- Can you also try this out and/or take a look at the description to get a general idea of how creating a manifest list will work? This is that "hairy" "third process which builds the manifest list," and while I do like the idea of pushing directly to an image tag by the builder with a new subcommand or flag, I'm not sure if that would be logistically feasible (please discuss!). Image builders would need permission to push to the repo the manifest lives in. Will we have a giant multi-arch repo that all fat manifests live in, or are all project (e.g. mongodb, golang, busybox) owners going to be responsible for collecting image refs for each multi-arch image and creating their own fat manifests?

Contributor

clnperez commented Dec 6, 2016

@stevvooe I updated the initial comment with a couple of flow examples. Is that helpful?

Honestly at this point I'd rather have a little more discussion around that flow before really having people digging into the code. Things there still need some work. I just wanted a working prototype for people to play with for now.

@tianon and @StefanScherer -- Can you also try this out and/or take a look at the description to get a general idea of how creating a manifest list will work? This is that "hairy" "third process which builds the manifest list," and while I do like the idea of pushing directly to an image tag by the builder with a new subcommand or flag, I'm not sure if that would be logistically feasible (please discuss!). Image builders would need permission to push to the repo the manifest lives in. Will we have a giant multi-arch repo that all fat manifests live in, or are all project (e.g. mongodb, golang, busybox) owners going to be responsible for collecting image refs for each multi-arch image and creating their own fat manifests?

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Dec 8, 2016

Contributor

@clnperez That looks great! Thank you so much for accommodating!

I think the missing aspect is that we need to represent the changes to the manifest in way that doesn't assume manifests can be modified in place. For example, annotating a manifest would changes its digest, creating a new manifest. With the cli flow presented here, that would entail changing the manifest for busybox with each change, which isn't ideal on a multi-user system. (Note that my suggestion contained this drawback).

There needs to be someway to represent the flow:

  1. I am creating a manifest list.
  2. I want to add these images, with these annotations.
  3. Now compile this manifest list.

Once that user flow is represented, I think we'll be a good spot.

Does that help? If this is still confusing, I can take another stab at the user flow.

Contributor

stevvooe commented Dec 8, 2016

@clnperez That looks great! Thank you so much for accommodating!

I think the missing aspect is that we need to represent the changes to the manifest in way that doesn't assume manifests can be modified in place. For example, annotating a manifest would changes its digest, creating a new manifest. With the cli flow presented here, that would entail changing the manifest for busybox with each change, which isn't ideal on a multi-user system. (Note that my suggestion contained this drawback).

There needs to be someway to represent the flow:

  1. I am creating a manifest list.
  2. I want to add these images, with these annotations.
  3. Now compile this manifest list.

Once that user flow is represented, I think we'll be a good spot.

Does that help? If this is still confusing, I can take another stab at the user flow.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Dec 8, 2016

Contributor

@stevvooe I do get what you're saying. I had the same concern when I started out, which is why my first (current) design was to put the annotated manifests into the user's ~/.docker/ in a new dir call manifests. However, that's not going to work for a couple of reasons (one of which is clear by the current jenkins failures). I'll work on an alternate solution.

Contributor

clnperez commented Dec 8, 2016

@stevvooe I do get what you're saying. I had the same concern when I started out, which is why my first (current) design was to put the annotated manifests into the user's ~/.docker/ in a new dir call manifests. However, that's not going to work for a couple of reasons (one of which is clear by the current jenkins failures). I'll work on an alternate solution.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Dec 16, 2016

Contributor

@clnperez Great!

It might be interesting to look into a model where the user downloads the manifest locally to a cache, makes the modifications, then posts it back.

I'll be interested in hearing what you come up with!

Contributor

stevvooe commented Dec 16, 2016

@clnperez Great!

It might be interesting to look into a model where the user downloads the manifest locally to a cache, makes the modifications, then posts it back.

I'll be interested in hearing what you come up with!

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Dec 16, 2016

Contributor

@stevvooe That's kind of what I was going for with putting the manifests in ~/.docker/manifests, but it looks like it's not always going to exist, like in the case of root. I could just have it create the ~/.docker dir if it doesn't exist. WDYT?

Contributor

clnperez commented Dec 16, 2016

@stevvooe That's kind of what I was going for with putting the manifests in ~/.docker/manifests, but it looks like it's not always going to exist, like in the case of root. I could just have it create the ~/.docker dir if it doesn't exist. WDYT?

@dquintela dquintela referenced this pull request Jan 6, 2017

Open

Create multi-arch image #6

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Jan 7, 2017

Contributor

@clnperez That sounds fine!

Contributor

stevvooe commented Jan 7, 2017

@clnperez That sounds fine!

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez Jan 9, 2017

Contributor

@stevvooe Awesome, thanks! I changed mkdir to mkdirAll and also fixed up a lot of newly-missing registry and digest stuff with a rebase. So now the tests pass, and I'll work on some of the "todos." Do you think we could shoot for getting this into 1.14?

Contributor

clnperez commented Jan 9, 2017

@stevvooe Awesome, thanks! I changed mkdir to mkdirAll and also fixed up a lot of newly-missing registry and digest stuff with a rebase. So now the tests pass, and I'll work on some of the "todos." Do you think we could shoot for getting this into 1.14?

@clnperez clnperez closed this May 3, 2017

@clnperez clnperez reopened this May 3, 2017

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 3, 2017

Contributor

Disregard that accidental close and half-comment. Wrong button!

Contributor

clnperez commented May 3, 2017

Disregard that accidental close and half-comment. Wrong button!

@xiekeyang

This comment has been minimized.

Show comment
Hide comment
@xiekeyang

xiekeyang May 4, 2017

Contributor

@clnperez I use manifest inspect command and encounter a problem. It can inspect a image with default(latest) tag. But when inspecting other tags, it prompt the error:

$ docker tag 136c8b16df20 localhost:5000/test:v1.0
$ docker push localhost:5000/test:v1.0
The push refers to a repository [localhost:5000/test]
v1.0: digest: sha256:5fd7abca95981a1ef978474f642f60dc16b1839fa41464d7382f6aa65e1309f9 size: 1364
$ docker manifest inspect localhost:5000/test:v1.0
ERRO[0000] Not continuing with pull after error: Error: image test not found 
FATA[0000] Error: image test not found                  

If it is a defect, or my fault of execution? The compile the binary based on your branch. The registry is that I run locally based on version of 2.6.1.
Thanks so much!

Contributor

xiekeyang commented May 4, 2017

@clnperez I use manifest inspect command and encounter a problem. It can inspect a image with default(latest) tag. But when inspecting other tags, it prompt the error:

$ docker tag 136c8b16df20 localhost:5000/test:v1.0
$ docker push localhost:5000/test:v1.0
The push refers to a repository [localhost:5000/test]
v1.0: digest: sha256:5fd7abca95981a1ef978474f642f60dc16b1839fa41464d7382f6aa65e1309f9 size: 1364
$ docker manifest inspect localhost:5000/test:v1.0
ERRO[0000] Not continuing with pull after error: Error: image test not found 
FATA[0000] Error: image test not found                  

If it is a defect, or my fault of execution? The compile the binary based on your branch. The registry is that I run locally based on version of 2.6.1.
Thanks so much!

Show outdated Hide outdated cli/command/manifest/inspect.go
// For now, always pull as there' no reason to store an inspect. They're quick to get.
// When the engine is multi-arch image aware, we can store these in a universal location to
// save a little bandwidth.
imgInspect, _, err = getImageData(dockerCli, named.Name(), "", true)

This comment has been minimized.

@xiekeyang

xiekeyang May 4, 2017

Contributor

It might should input named.String() here, otherwise will loss the tag.

@xiekeyang

xiekeyang May 4, 2017

Contributor

It might should input named.String() here, otherwise will loss the tag.

This comment has been minimized.

@clnperez

clnperez May 4, 2017

Contributor

👍 I thought for sure we had tests for that but I'm adding one, and adding this fix as well. Thanks @xiekeyang!

@clnperez

clnperez May 4, 2017

Contributor

👍 I thought for sure we had tests for that but I'm adding one, and adding this fix as well. Thanks @xiekeyang!

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer May 5, 2017

Contributor

I just used the Docker swarm constraints node.platform.os == linux and node.platform.os == windows to run Faas in a cross-platform swarm.
I checked if there is a node.platform.arch - yes it is. But the example in docker/swarmkit#1678 shows node.platform.arch == x86_64
The manifest-tool uses eg. arch = amd64.
Is this only a typo in docker/swarmkit#1678 or do we have to think of two different values?

Contributor

StefanScherer commented May 5, 2017

I just used the Docker swarm constraints node.platform.os == linux and node.platform.os == windows to run Faas in a cross-platform swarm.
I checked if there is a node.platform.arch - yes it is. But the example in docker/swarmkit#1678 shows node.platform.arch == x86_64
The manifest-tool uses eg. arch = amd64.
Is this only a typo in docker/swarmkit#1678 or do we have to think of two different values?

@aaronlehmann

This comment has been minimized.

Show comment
Hide comment
@aaronlehmann

aaronlehmann May 5, 2017

Contributor

I believe that example is misleading.

Contributor

aaronlehmann commented May 5, 2017

I believe that example is misleading.

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer May 5, 2017

Contributor

It seems that these two different Architecture names are already in use:

$ docker image inspect alpine | grep Arch
        "Architecture": "amd64",

$ docker node inspect moby | grep Arch
                "Architecture": "x86_64",

So images use amd64, nodes x86_64.

So this docker-compose.yml also does not work as the replicas stay at 0/1:

version: "3"
services:
    whoami:
        image: stefanscherer/whoami
        deploy:
            placement:
                constraints: [node.platform.arch == amd64]
Contributor

StefanScherer commented May 5, 2017

It seems that these two different Architecture names are already in use:

$ docker image inspect alpine | grep Arch
        "Architecture": "amd64",

$ docker node inspect moby | grep Arch
                "Architecture": "x86_64",

So images use amd64, nodes x86_64.

So this docker-compose.yml also does not work as the replicas stay at 0/1:

version: "3"
services:
    whoami:
        image: stefanscherer/whoami
        deploy:
            placement:
                constraints: [node.platform.arch == amd64]

@StefanScherer StefanScherer referenced this pull request May 5, 2017

Closed

Add windows function #79

1 of 1 task complete

@thaJeztah thaJeztah added the impact/cli label May 5, 2017

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 8, 2017

Contributor

@stevvooe Here's the new inspect output (a real manifest object) of a single manifest:

 ./bundles/latest/binary-client/docker-17.06.0-dev manifest inspect busybox
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1465,
        "digest": "sha256:00f017a8c2a6e1fe2ffd05c281f27d069d2a99323a8cd514dd35f228ba26d2ff"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 677607,
            "digest": "sha256:7520415ce76232cdd62ecc345cea5ea44f5b6b144dc62351f2cd2b08382532a3"
        }
    ]
}

Also in case people want to know the arch, etc., for finding images for a manifest list, etc., I added a -p (for platform) flag for the single manifests. I think this goes nicely with the "shallow pull" usage scenario. WDYT?

> ./bundles/latest/binary-client/docker-17.06.0-dev manifest inspect -p busybox 
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1465,
        "digest": "sha256:00f017a8c2a6e1fe2ffd05c281f27d069d2a99323a8cd514dd35f228ba26d2ff"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 677607,
            "digest": "sha256:7520415ce76232cdd62ecc345cea5ea44f5b6b144dc62351f2cd2b08382532a3"
        }
    ]
}
{
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "size": 527,
    "digest": "sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f",
    "platform": {
        "architecture": "amd64",
        "os": "linux"
    }
}

Also -- for the manifest lists, here's the output now:

> ./bundles/latest/binary-client/docker-17.06.0-dev manifest inspect clnperez/hello-world
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 524,
            "digest": "sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7",
            "platform": {
                "architecture": "arm",
                "os": "freebsd"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 525,
            "digest": "sha256:7d57adf137665f748956c86089320710b66d08584db3500ed98f4bb3da637c2d",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 527,
            "digest": "sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f",
            "platform": {
                "architecture": "arm",
                "os": "freebsd"
            }
        }
    ]
}

I'm not a huge fan of having just this. Compared to before (and @estesp's tool) the layers aren't shown any more. Is there a way for a user to trace just a digest back to an image? IOW, having that docker manifest list inspect output, could someone do a docker manifest inspect XYZ and see the layers? Or should I also have a flag that prints add'l info like layers?

Contributor

clnperez commented May 8, 2017

@stevvooe Here's the new inspect output (a real manifest object) of a single manifest:

 ./bundles/latest/binary-client/docker-17.06.0-dev manifest inspect busybox
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1465,
        "digest": "sha256:00f017a8c2a6e1fe2ffd05c281f27d069d2a99323a8cd514dd35f228ba26d2ff"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 677607,
            "digest": "sha256:7520415ce76232cdd62ecc345cea5ea44f5b6b144dc62351f2cd2b08382532a3"
        }
    ]
}

Also in case people want to know the arch, etc., for finding images for a manifest list, etc., I added a -p (for platform) flag for the single manifests. I think this goes nicely with the "shallow pull" usage scenario. WDYT?

> ./bundles/latest/binary-client/docker-17.06.0-dev manifest inspect -p busybox 
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1465,
        "digest": "sha256:00f017a8c2a6e1fe2ffd05c281f27d069d2a99323a8cd514dd35f228ba26d2ff"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 677607,
            "digest": "sha256:7520415ce76232cdd62ecc345cea5ea44f5b6b144dc62351f2cd2b08382532a3"
        }
    ]
}
{
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "size": 527,
    "digest": "sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f",
    "platform": {
        "architecture": "amd64",
        "os": "linux"
    }
}

Also -- for the manifest lists, here's the output now:

> ./bundles/latest/binary-client/docker-17.06.0-dev manifest inspect clnperez/hello-world
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 524,
            "digest": "sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7",
            "platform": {
                "architecture": "arm",
                "os": "freebsd"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 525,
            "digest": "sha256:7d57adf137665f748956c86089320710b66d08584db3500ed98f4bb3da637c2d",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 527,
            "digest": "sha256:32f093055929dbc23dec4d03e09dfe971f5973a9ca5cf059cbfb644c206aa83f",
            "platform": {
                "architecture": "arm",
                "os": "freebsd"
            }
        }
    ]
}

I'm not a huge fan of having just this. Compared to before (and @estesp's tool) the layers aren't shown any more. Is there a way for a user to trace just a digest back to an image? IOW, having that docker manifest list inspect output, could someone do a docker manifest inspect XYZ and see the layers? Or should I also have a flag that prints add'l info like layers?

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe May 9, 2017

Contributor

@clnperez For the most part, the layers are an implementation detail. If one wants to inspect the layers, they should be able to inspect any sub-manifest that is referenced in the manifest list.

That said, it seems like you can make a "sugary" out that meets the goal without having that impact the internal data structures.

Contributor

stevvooe commented May 9, 2017

@clnperez For the most part, the layers are an implementation detail. If one wants to inspect the layers, they should be able to inspect any sub-manifest that is referenced in the manifest list.

That said, it seems like you can make a "sugary" out that meets the goal without having that impact the internal data structures.

@xiekeyang

This comment has been minimized.

Show comment
Hide comment
@xiekeyang

xiekeyang May 12, 2017

Contributor

@clnperez I encounter a problem when pushing manifest yaml file.

$ docker -D manifest push -f manifest.yaml
Error setting up repository endpoint and references for "ihub.com/user/busybox:latest": open /etc/docker/certs.d/ihub.com: permission denied

The permission is denied even after login. And I've added ihub.com to insecure-registry.
In this case I'm no-root user.

Contributor

xiekeyang commented May 12, 2017

@clnperez I encounter a problem when pushing manifest yaml file.

$ docker -D manifest push -f manifest.yaml
Error setting up repository endpoint and references for "ihub.com/user/busybox:latest": open /etc/docker/certs.d/ihub.com: permission denied

The permission is denied even after login. And I've added ihub.com to insecure-registry.
In this case I'm no-root user.

@xiekeyang

This comment has been minimized.

Show comment
Hide comment
@xiekeyang

xiekeyang May 12, 2017

Contributor

It seems that the transfer of options of bearer token and insecure registries need to be fixed.

Contributor

xiekeyang commented May 12, 2017

It seems that the transfer of options of bearer token and insecure registries need to be fixed.

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas May 12, 2017

@clnperez Could you please post a TL;DR; of the current patch? Sum up what the UX will look like, what the commands will do on a high level and so forth?

Reviewing this line-by-line is hard until you know what this is all about on a high level. I know exactly how estesp/manifest-tool works as I've read all the code there, but IIUC this PR has evolved using that only as a starting point.

Thanks, I want to see this mainlined as soon as possible and want to help to review it!

luxas commented May 12, 2017

@clnperez Could you please post a TL;DR; of the current patch? Sum up what the UX will look like, what the commands will do on a high level and so forth?

Reviewing this line-by-line is hard until you know what this is all about on a high level. I know exactly how estesp/manifest-tool works as I've read all the code there, but IIUC this PR has evolved using that only as a starting point.

Thanks, I want to see this mainlined as soon as possible and want to help to review it!

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 12, 2017

Contributor

@xiekeyang In order to use insecure registries with this command (since under the covers it has no association whatsoever with the engine), I added a new option to the local (user-scoped) config file. See https://github.com/clnperez/docker/blob/8d2a1422ce906d2a47fe20d3ee42eec6e31881b9/cli/config/configfile/file.go#L43.

In ~/.docker/config.json you would do something like:

{                                                                                                                                                     
    "auths": {                                                                                                                                        
        "https://index.docker.io/v1/": {                                                                                                              
            "auth": "myauthtokenhere"                                                                                                        
        }                                                                                                                                             
    },                                                                                                                                                
    "insecure-registries" : ["ihub.com:5000"]                                                                                                        
}                                                                                                                                                     
Contributor

clnperez commented May 12, 2017

@xiekeyang In order to use insecure registries with this command (since under the covers it has no association whatsoever with the engine), I added a new option to the local (user-scoped) config file. See https://github.com/clnperez/docker/blob/8d2a1422ce906d2a47fe20d3ee42eec6e31881b9/cli/config/configfile/file.go#L43.

In ~/.docker/config.json you would do something like:

{                                                                                                                                                     
    "auths": {                                                                                                                                        
        "https://index.docker.io/v1/": {                                                                                                              
            "auth": "myauthtokenhere"                                                                                                        
        }                                                                                                                                             
    },                                                                                                                                                
    "insecure-registries" : ["ihub.com:5000"]                                                                                                        
}                                                                                                                                                     
@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 12, 2017

Contributor

@StefanScherer That node constraints vs image arch discrepancy is unfortunate. On my system the uname -m ouput is x86_64 but GOARCH="amd64". Maybe this should be a change in swarm (to use amd64, since go does, and that's what docker currently uses as well)? As a workaround can you just add an or condition to your node constraints?

Contributor

clnperez commented May 12, 2017

@StefanScherer That node constraints vs image arch discrepancy is unfortunate. On my system the uname -m ouput is x86_64 but GOARCH="amd64". Maybe this should be a change in swarm (to use amd64, since go does, and that's what docker currently uses as well)? As a workaround can you just add an or condition to your node constraints?

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 12, 2017

Contributor

@luxas Does reading the commit msg (buried by this point in time) help?
f52aa60

Contributor

clnperez commented May 12, 2017

@luxas Does reading the commit msg (buried by this point in time) help?
f52aa60

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer May 13, 2017

Contributor

After building the binaries of this PR I found a problem when I just run docker manifest push without any more parameter, just wanted to see the output/usage message of that command:

docker@v17:~$ docker manifest push
panic: runtime error: invalid memory address or nil pointer dereference

Then I tried to understand how to use the manifest commands with an image I already have built with the manifest-tool. So here is what I understand with the usage and the commit message given.

With the manifest-tool I have these two commands to create the "1.7.1" tag and the "latest" tag (both at Docker Hub stefanscherer/winspector/tags)

$ manifest-tool push from-args --platforms linux/amd64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspector:1.7.1
$ manifest-tool push from-args --platforms linux/amd64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspector:latest

So as a Docker Captain I tried to do steer my ship into unexplored waters. 😄
My first thought was I have to create a yaml or the create command creates a yaml for me, so I looked for a test.yml file after running docker manifest create test.yml stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1. Then I found out that these files are at ~/.docker/manifests. Haven't expected that there. Do I pollute my Docker build server with outdated files (eg from previous builds) there?

Is the new-list-ref-name the name for the later place at Docker Hub? So I tried to do this using a tag "1.7.1-test" for my multi-os image:

$ docker manifest create stefanscherer/winspector:1.7.1-test stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
$ ls .docker/manifests/docker.io_stefanscherer_winspector-latest/

Seems like this does not work and a "latest" will be used instead. So can we create "tags" other than latest?

So for the further test I created a new repo at Docker Hub to push into a safe place.

$ docker manifest create stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
$ docker manifest push stefanscherer/winspectortest

Next try to annotate more:

$ docker manifest annotate stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 --os linux --arch amd64
$ docker manifest annotate stefanscherer/winspectortest stefanscherer/winspector:windows-1.7.1 --os windows --arch amd64
$ docker manifest push stefanscherer/winspectortest
INFO[0000] Retrieving digests of images...              
INFO[0000] Image "docker.io/stefanscherer/winspector:windows-1.7.1" is digest sha256:6b2e250422e969702509200254d075c8d1c0b88f09b76d10a33043e3e35bbe41; size: 3656 
INFO[0000] Image "docker.io/stefanscherer/winspector:linux-1.7.1" is digest sha256:6097f83a1b6f02f504b4a6fbb1f5e582b06b597add10139f7f46f6a4001c8420; size: 1994 
INFO[0000] manifest: put: target endpoint url: https://registry-1.docker.io 
Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202
Contributor

StefanScherer commented May 13, 2017

After building the binaries of this PR I found a problem when I just run docker manifest push without any more parameter, just wanted to see the output/usage message of that command:

docker@v17:~$ docker manifest push
panic: runtime error: invalid memory address or nil pointer dereference

Then I tried to understand how to use the manifest commands with an image I already have built with the manifest-tool. So here is what I understand with the usage and the commit message given.

With the manifest-tool I have these two commands to create the "1.7.1" tag and the "latest" tag (both at Docker Hub stefanscherer/winspector/tags)

$ manifest-tool push from-args --platforms linux/amd64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspector:1.7.1
$ manifest-tool push from-args --platforms linux/amd64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspector:latest

So as a Docker Captain I tried to do steer my ship into unexplored waters. 😄
My first thought was I have to create a yaml or the create command creates a yaml for me, so I looked for a test.yml file after running docker manifest create test.yml stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1. Then I found out that these files are at ~/.docker/manifests. Haven't expected that there. Do I pollute my Docker build server with outdated files (eg from previous builds) there?

Is the new-list-ref-name the name for the later place at Docker Hub? So I tried to do this using a tag "1.7.1-test" for my multi-os image:

$ docker manifest create stefanscherer/winspector:1.7.1-test stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
$ ls .docker/manifests/docker.io_stefanscherer_winspector-latest/

Seems like this does not work and a "latest" will be used instead. So can we create "tags" other than latest?

So for the further test I created a new repo at Docker Hub to push into a safe place.

$ docker manifest create stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
$ docker manifest push stefanscherer/winspectortest

Next try to annotate more:

$ docker manifest annotate stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 --os linux --arch amd64
$ docker manifest annotate stefanscherer/winspectortest stefanscherer/winspector:windows-1.7.1 --os windows --arch amd64
$ docker manifest push stefanscherer/winspectortest
INFO[0000] Retrieving digests of images...              
INFO[0000] Image "docker.io/stefanscherer/winspector:windows-1.7.1" is digest sha256:6b2e250422e969702509200254d075c8d1c0b88f09b76d10a33043e3e35bbe41; size: 3656 
INFO[0000] Image "docker.io/stefanscherer/winspector:linux-1.7.1" is digest sha256:6097f83a1b6f02f504b4a6fbb1f5e582b06b597add10139f7f46f6a4001c8420; size: 1994 
INFO[0000] manifest: put: target endpoint url: https://registry-1.docker.io 
Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202
@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 15, 2017

Contributor

@StefanScherer, a gigantic thanks for steering your ship off into these uncharted waters. :D

I'll try to address all the items from your comment...

  • As for your push error, I don't get that, which is very strange:
./bundles/latest/binary-client/docker-17.06.0-dev manifest push
Please push using a yaml file or a list created using 'manifest create.'
  • You do have to manually create the yaml file from scratch (although, a command to create a yaml file for later use after all your annotations have been done once sounds like a really useful follow-on feature). So, I definitely need to update the commit message to be more helpful. Sorry for this confusion!

  • The create command does create the files you found, and I am going to add a flag to clean those up after your push completes, so the dirtying is a temporary badness. :)

  • And yes, the new-list-ref-name is the name for the later place at Docker Hub.

  • The create command should have pushed tags, so I think you found a bug. I'll look at that.

  • Just FYI, the annotations for linux/amd64 aren't necessary (but my example does list them so it seems like they are).

Had you done a docker login before you ran the docker manifest push command?

Let me know if I missed or misunderstood anything here, and thanks again for testing.

Contributor

clnperez commented May 15, 2017

@StefanScherer, a gigantic thanks for steering your ship off into these uncharted waters. :D

I'll try to address all the items from your comment...

  • As for your push error, I don't get that, which is very strange:
./bundles/latest/binary-client/docker-17.06.0-dev manifest push
Please push using a yaml file or a list created using 'manifest create.'
  • You do have to manually create the yaml file from scratch (although, a command to create a yaml file for later use after all your annotations have been done once sounds like a really useful follow-on feature). So, I definitely need to update the commit message to be more helpful. Sorry for this confusion!

  • The create command does create the files you found, and I am going to add a flag to clean those up after your push completes, so the dirtying is a temporary badness. :)

  • And yes, the new-list-ref-name is the name for the later place at Docker Hub.

  • The create command should have pushed tags, so I think you found a bug. I'll look at that.

  • Just FYI, the annotations for linux/amd64 aren't necessary (but my example does list them so it seems like they are).

Had you done a docker login before you ran the docker manifest push command?

Let me know if I missed or misunderstood anything here, and thanks again for testing.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 16, 2017

Contributor

@tiborvass Should I bite the bullet and move this over to the new cli repo? Though a little sad to lose the history, it's maybe good to refocus the discussion (maybe with Stefan's questions) since I believe the command design is settled at this point.

Contributor

clnperez commented May 16, 2017

@tiborvass Should I bite the bullet and move this over to the new cli repo? Though a little sad to lose the history, it's maybe good to refocus the discussion (maybe with Stefan's questions) since I believe the command design is settled at this point.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 16, 2017

Contributor

@StefanScherer I fixed the tags bug, and also added a -p option on push (which is true by default) to delete those files after a successful manifest list push.

Contributor

clnperez commented May 16, 2017

@StefanScherer I fixed the tags bug, and also added a -p option on push (which is true by default) to delete those files after a successful manifest list push.

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer May 17, 2017

Contributor

Thanks @clnperez. I've fetched the PR again and now cross-compiled an OSX client binary.
Small issue here:

$ bundles/latest/cross/darwin/amd64/docker manifest create stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
INFO[0000] Retrieving digests of images...              
ERRO[0004] Error storing manifests: homedir.GetStatic() is not supported on this system
 
ERRO[0009] Error storing manifests: homedir.GetStatic() is not supported on this system

So I'll test with a Windows client, but the same here:

PS C:\Users\stefan\code\dockerfiles-windows\docker\binary> .\docker.exe manifest create stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
time="2017-05-17T06:14:37-07:00" level=info msg="Retrieving digests of images..."
time="2017-05-17T06:14:40-07:00" level=error msg="Error storing manifests: homedir.GetStatic() is not supported on this system\n"
time="2017-05-17T06:14:45-07:00" level=error msg="Error storing manifests: homedir.GetStatic() is not supported on this system\n"

Changing cli/command/manifest/push.go and cli/command/manifest/util.go to homedir.Get() solves the problem on OSX and Windows.

So I tried it again with a single repository:

$ bundles/latest/cross/darwin/amd64/docker manifest create stefanscherer/winspector:1.7.1-test-pr27455 stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
$ bundles/latest/cross/darwin/amd64/docker manifest push stefanscherer/winspector:1.7.1-test-pr27455

That looks good and is pretty simple to use as well. 👍

Only thing I still have an issue pushing it to another repository

$ bundles/latest/cross/darwin/amd64/docker manifest create stefanscherer/winspectortest:1.7.1 stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
INFO[0000] Retrieving digests of images...              
$ bundles/latest/cross/darwin/amd64/docker manifest push stefanscherer/winspectortest:1.7.1
INFO[0000] Retrieving digests of images...              
INFO[0000] manifest: put: target endpoint url: https://registry-1.docker.io 
Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202

The docker client is logged in and has the rights to pull from and push into both repositories used here.

Contributor

StefanScherer commented May 17, 2017

Thanks @clnperez. I've fetched the PR again and now cross-compiled an OSX client binary.
Small issue here:

$ bundles/latest/cross/darwin/amd64/docker manifest create stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
INFO[0000] Retrieving digests of images...              
ERRO[0004] Error storing manifests: homedir.GetStatic() is not supported on this system
 
ERRO[0009] Error storing manifests: homedir.GetStatic() is not supported on this system

So I'll test with a Windows client, but the same here:

PS C:\Users\stefan\code\dockerfiles-windows\docker\binary> .\docker.exe manifest create stefanscherer/winspectortest stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
time="2017-05-17T06:14:37-07:00" level=info msg="Retrieving digests of images..."
time="2017-05-17T06:14:40-07:00" level=error msg="Error storing manifests: homedir.GetStatic() is not supported on this system\n"
time="2017-05-17T06:14:45-07:00" level=error msg="Error storing manifests: homedir.GetStatic() is not supported on this system\n"

Changing cli/command/manifest/push.go and cli/command/manifest/util.go to homedir.Get() solves the problem on OSX and Windows.

So I tried it again with a single repository:

$ bundles/latest/cross/darwin/amd64/docker manifest create stefanscherer/winspector:1.7.1-test-pr27455 stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
$ bundles/latest/cross/darwin/amd64/docker manifest push stefanscherer/winspector:1.7.1-test-pr27455

That looks good and is pretty simple to use as well. 👍

Only thing I still have an issue pushing it to another repository

$ bundles/latest/cross/darwin/amd64/docker manifest create stefanscherer/winspectortest:1.7.1 stefanscherer/winspector:linux-1.7.1 stefanscherer/winspector:windows-1.7.1
INFO[0000] Retrieving digests of images...              
$ bundles/latest/cross/darwin/amd64/docker manifest push stefanscherer/winspectortest:1.7.1
INFO[0000] Retrieving digests of images...              
INFO[0000] manifest: put: target endpoint url: https://registry-1.docker.io 
Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202

The docker client is logged in and has the rights to pull from and push into both repositories used here.

@StefanScherer

Marked the two places to use homedir.Get() to make it work on non-Linux platforms.

Show outdated Hide outdated cli/command/manifest/util.go
if err := ensureHomeIfIAmStatic(); err != nil {
return "", err
}
userHome, err := homedir.GetStatic()

This comment has been minimized.

@StefanScherer

StefanScherer May 17, 2017

Contributor

Change to userHome := homedir.Get()

@StefanScherer

StefanScherer May 17, 2017

Contributor

Change to userHome := homedir.Get()

This comment has been minimized.

@clnperez

clnperez May 17, 2017

Contributor

What if the user's homedir isn't set? I don't think we can use this just in case it isn't.

@clnperez

clnperez May 17, 2017

Contributor

What if the user's homedir isn't set? I don't think we can use this just in case it isn't.

This comment has been minimized.

@clnperez

clnperez May 17, 2017

Contributor

I take that back. Let me take another look at an alternative. Can you try compiling a dynamic binary?

@clnperez

clnperez May 17, 2017

Contributor

I take that back. Let me take another look at an alternative. Can you try compiling a dynamic binary?

This comment has been minimized.

This comment has been minimized.

@clnperez

clnperez May 17, 2017

Contributor

Sorry for the bajillionth comment on this but @StefanScherer, is the home dir always going to be set on windows? I didn't realize that there was a different user package in runc/libcontainer that gets us around the homedir segfault problem with static binaries. But it does return an empty string on windows.

@clnperez

clnperez May 17, 2017

Contributor

Sorry for the bajillionth comment on this but @StefanScherer, is the home dir always going to be set on windows? I didn't realize that there was a different user package in runc/libcontainer that gets us around the homedir segfault problem with static binaries. But it does return an empty string on windows.

Show outdated Hide outdated cli/command/manifest/push.go
insecureRegistries := []string{}
// Check $HOME/.docker/config.json. There may be mismatches between what the user has in their
// local config and what the daemon they're talking to allows, but we can be okay with that.
userHome, err := homedir.GetStatic()

This comment has been minimized.

@StefanScherer

StefanScherer May 17, 2017

Contributor

Change to userHome := homedir.Get()

@StefanScherer

StefanScherer May 17, 2017

Contributor

Change to userHome := homedir.Get()

This comment has been minimized.

@clnperez

clnperez May 17, 2017

Contributor

same :)

@clnperez

clnperez May 17, 2017

Contributor

same :)

This comment has been minimized.

@StefanScherer

StefanScherer May 17, 2017

Contributor

The same function is used to read the config.json for Docker Hub credentials, so I think this is fine. On Windows I do not have HOME set and it worked.

@StefanScherer

StefanScherer May 17, 2017

Contributor

The same function is used to read the config.json for Docker Hub credentials, so I think this is fine. On Windows I do not have HOME set and it worked.

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp May 17, 2017

Contributor

Definitely confused on @StefanScherer's log regarding cross-repo mount. My manifest-tool code also expects 201 (Created), but it looks like the registry code has been returning 202 (Accepted) for well over 2 years.

So now I'm not sure why manifest-tool doesn't error out for the same reason?

Contributor

estesp commented May 17, 2017

Definitely confused on @StefanScherer's log regarding cross-repo mount. My manifest-tool code also expects 201 (Created), but it looks like the registry code has been returning 202 (Accepted) for well over 2 years.

So now I'm not sure why manifest-tool doesn't error out for the same reason?

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer May 17, 2017

Contributor

@estesp I never tried cross-repository pushes with manifest-tool. I only did that to test this PR when it only supported creating a "latest" and I didn't want to overwrite it.

Using the same cross-repo push with manifest-tool shows the same error:

$ ./manifest-tool-darwin-amd64 push from-args --platforms linux/amd64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspectortest:latest
INFO[0000] Retrieving digests of images...              
INFO[0004] Image "stefanscherer/winspector:linux-1.7.1" is digest sha256:6097f83a1b6f02f504b4a6fbb1f5e582b06b597add10139f7f46f6a4001c8420; size: 1994 
INFO[0008] Image "stefanscherer/winspector:windows-1.7.1" is digest sha256:6b2e250422e969702509200254d075c8d1c0b88f09b76d10a33043e3e35bbe41; size: 3656 
FATA[0021] Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202 
Contributor

StefanScherer commented May 17, 2017

@estesp I never tried cross-repository pushes with manifest-tool. I only did that to test this PR when it only supported creating a "latest" and I didn't want to overwrite it.

Using the same cross-repo push with manifest-tool shows the same error:

$ ./manifest-tool-darwin-amd64 push from-args --platforms linux/amd64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspectortest:latest
INFO[0000] Retrieving digests of images...              
INFO[0004] Image "stefanscherer/winspector:linux-1.7.1" is digest sha256:6097f83a1b6f02f504b4a6fbb1f5e582b06b597add10139f7f46f6a4001c8420; size: 1994 
INFO[0008] Image "stefanscherer/winspector:windows-1.7.1" is digest sha256:6b2e250422e969702509200254d075c8d1c0b88f09b76d10a33043e3e35bbe41; size: 3656 
FATA[0021] Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202 
@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp May 17, 2017

Contributor

@StefanScherer would love to get to the bottom of this; seems easy to just change the code to accept either success code (201 or 202), but I'd like to understand why :)

I just tested what I think is a very similar setup inside my own DockerHub repo and I can't recreate the error. Would you mind running your same command with --debug in your manifest-tool invocation? I'd like to see what is different as my cross-repo blob mounts work properly.

Contributor

estesp commented May 17, 2017

@StefanScherer would love to get to the bottom of this; seems easy to just change the code to accept either success code (201 or 202), but I'd like to understand why :)

I just tested what I think is a very similar setup inside my own DockerHub repo and I can't recreate the error. Would you mind running your same command with --debug in your manifest-tool invocation? I'd like to see what is different as my cross-repo blob mounts work properly.

@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 17, 2017

Contributor

@estesp The private repo used in the integration tests does the cross-repo blob mounts, and those pass. 🤔

Contributor

clnperez commented May 17, 2017

@estesp The private repo used in the integration tests does the cross-repo blob mounts, and those pass. 🤔

@StefanScherer

This comment has been minimized.

Show comment
Hide comment
@StefanScherer

StefanScherer May 17, 2017

Contributor

I use a Windows Docker image in that example and it is the foreign Windows base layer that causes the problem:

PS C:\Windows\system32> C:\Users\stefan\code\docker\manifest-tool-windows-amd64.exe --debug push from-args --platforms linux/amd
64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspectortest:latest
time="2017-05-17T13:09:09-07:00" level=debug msg="endpoints: [{false https://registry-1.docker.io v2 true true 0xc0422d8000}]"
time="2017-05-17T13:09:09-07:00" level=debug msg="repoName: stefanscherer/winspectortest"
time="2017-05-17T13:09:09-07:00" level=info msg="Retrieving digests of images..."
time="2017-05-17T13:09:09-07:00" level=debug msg="authConfig for docker.io: stefscherer"
time="2017-05-17T13:09:09-07:00" level=debug msg="endpoints: [{false https://registry-1.docker.io v2 true true 0xc0422d8420}]"
time="2017-05-17T13:09:09-07:00" level=debug msg="attempting v1 ping for registry endpoint https://registry-1.docker.io/v1/"
time="2017-05-17T13:09:10-07:00" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
time="2017-05-17T13:09:10-07:00" level=debug msg="Registry version header: '0.8.15'"
time="2017-05-17T13:09:10-07:00" level=debug msg="PingResult.Version: \"0.8.15\""
time="2017-05-17T13:09:10-07:00" level=debug msg="Registry standalone header: ''"
time="2017-05-17T13:09:10-07:00" level=debug msg="PingResult.Standalone: true"
time="2017-05-17T13:09:10-07:00" level=debug msg="Trying to fetch image manifest of stefanscherer/winspector repository from https://registry-1.docker.io v2"
time="2017-05-17T13:09:15-07:00" level=info msg="Image \"stefanscherer/winspector:linux-1.7.1\" is digest sha256:6097f83a1b6f02f504b4a6fbb1f5e582b06b597add10139f7f46f6a4001c8420; size: 1994"
time="2017-05-17T13:09:15-07:00" level=debug msg="Adding manifest references of \"stefanscherer/winspector:linux-1.7.1\" to blob mount requests"
time="2017-05-17T13:09:15-07:00" level=debug msg="Adding manifest \"stefanscherer/winspector\" -> to be pushed to \"stefanscherer/winspectortest\" as a manifest reference"
time="2017-05-17T13:09:15-07:00" level=debug msg="authConfig for docker.io: stefscherer"
time="2017-05-17T13:09:15-07:00" level=debug msg="endpoints: [{false https://registry-1.docker.io v2 true true 0xc0422d9340}]"
time="2017-05-17T13:09:15-07:00" level=debug msg="attempting v1 ping for registry endpoint https://registry-1.docker.io/v1/"
time="2017-05-17T13:09:17-07:00" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
time="2017-05-17T13:09:17-07:00" level=debug msg="Registry version header: '0.8.15'"
time="2017-05-17T13:09:17-07:00" level=debug msg="PingResult.Version: \"0.8.15\""
time="2017-05-17T13:09:17-07:00" level=debug msg="Registry standalone header: ''"
time="2017-05-17T13:09:17-07:00" level=debug msg="PingResult.Standalone: true"
time="2017-05-17T13:09:17-07:00" level=debug msg="Trying to fetch image manifest of stefanscherer/winspector repository from https://registry-1.docker.io v2"
time="2017-05-17T13:09:22-07:00" level=info msg="Image \"stefanscherer/winspector:windows-1.7.1\" is digest sha256:6b2e250422e969702509200254d075c8d1c0b88f09b76d10a33043e3e35bbe41; size: 3656"
time="2017-05-17T13:09:22-07:00" level=debug msg="Adding manifest references of \"stefanscherer/winspector:windows-1.7.1\" to blob mount requests"
time="2017-05-17T13:09:22-07:00" level=debug msg="Adding manifest \"stefanscherer/winspector\" -> to be pushed to \"stefanscherer/winspectortest\" as a manifest reference"
time="2017-05-17T13:09:22-07:00" level=debug msg="Manifest list push url: https://registry-1.docker.io/v2/stefanscherer/winspectortest/manifests/latest"
time="2017-05-17T13:09:22-07:00" level=debug msg="mediaType of manifestList: application/vnd.docker.distribution.manifest.list.v2+json"
time="2017-05-17T13:09:22-07:00" level=debug msg="authConfig for docker.io: stefscherer"
time="2017-05-17T13:09:25-07:00" level=debug msg="Mount of blob sha256:ddbc94156cc3298fd4235bf5d6d775496c4af1a631f8974bd3aa36d954ab81b5 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:ddbc94156cc3298fd4235bf5d6d775496c4af1a631f8974bd3aa36d954ab81b5\""
time="2017-05-17T13:09:27-07:00" level=debug msg="Mount of blob sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93\""
time="2017-05-17T13:09:30-07:00" level=debug msg="Mount of blob sha256:7f8ede2d2484a67ae3a88912d400c46f4f76e8f62f1003ed10c6f95893603781 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:7f8ede2d2484a67ae3a88912d400c46f4f76e8f62f1003ed10c6f95893603781\""
time="2017-05-17T13:09:31-07:00" level=debug msg="Mount of blob sha256:3c752c95ebfb440ae472bd7c28a85adde5844abc783e8e0c12585b3055e9e804 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:3c752c95ebfb440ae472bd7c28a85adde5844abc783e8e0c12585b3055e9e804\""
time="2017-05-17T13:09:33-07:00" level=debug msg="Mount of blob sha256:39c204c948870160e7e0aeaeaf01229492437a04a18345faed5e2733b0d0a64e succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:39c204c948870160e7e0aeaeaf01229492437a04a18345faed5e2733b0d0a64e\""
time="2017-05-17T13:09:35-07:00" level=debug msg="Mount of blob sha256:4d20e2a064f4cbd8241815318cdc2422f2b6ef6384e2af55892a440deca15f72 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:4d20e2a064f4cbd8241815318cdc2422f2b6ef6384e2af55892a440deca15f72\""
time="2017-05-17T13:09:37-07:00" level=debug msg="Mount of blob sha256:f63a734d587e2d53ab4390c2f6e8d586d70eb5995bebf707f395c24d600b8bd5 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:f63a734d587e2d53ab4390c2f6e8d586d70eb5995bebf707f395c24d600b8bd5\""
time="2017-05-17T13:09:38-07:00" level=debug msg="Mount of blob sha256:8b64818b6bdd13c5973a3e8e4a3996a1ff72cecabc29a62e28fa58568d437878 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:8b64818b6bdd13c5973a3e8e4a3996a1ff72cecabc29a62e28fa58568d437878\""
time="2017-05-17T13:09:40-07:00" level=debug msg="Mount of blob sha256:ae59d408cfc3ff127671707ec5580a1b35405e743f8022ca04cd2171cd651316 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:ae59d408cfc3ff127671707ec5580a1b35405e743f8022ca04cd2171cd651316\""
time="2017-05-17T13:09:42-07:00" level=debug msg="Mount of blob sha256:4de619810b10400a3ac26955f47b84d1794f50493f9a24a02d4ffd3b94c7de81 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:4de619810b10400a3ac26955f47b84d1794f50493f9a24a02d4ffd3b94c7de81\""
time="2017-05-17T13:09:45-07:00" level=fatal msg="Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202"

The layer with sha256:bce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277 is the microsoft/nanoserver:10.0.14393.447 base layer, there also is a second sha256:4a8c367fd46d2e2da2a8b0fa02158540e13b3a9015daf9f17d1af354a591492f which is the Windows update layer.

Contributor

StefanScherer commented May 17, 2017

I use a Windows Docker image in that example and it is the foreign Windows base layer that causes the problem:

PS C:\Windows\system32> C:\Users\stefan\code\docker\manifest-tool-windows-amd64.exe --debug push from-args --platforms linux/amd
64,windows/amd64 --template stefanscherer/winspector:OS-1.7.1 --target stefanscherer/winspectortest:latest
time="2017-05-17T13:09:09-07:00" level=debug msg="endpoints: [{false https://registry-1.docker.io v2 true true 0xc0422d8000}]"
time="2017-05-17T13:09:09-07:00" level=debug msg="repoName: stefanscherer/winspectortest"
time="2017-05-17T13:09:09-07:00" level=info msg="Retrieving digests of images..."
time="2017-05-17T13:09:09-07:00" level=debug msg="authConfig for docker.io: stefscherer"
time="2017-05-17T13:09:09-07:00" level=debug msg="endpoints: [{false https://registry-1.docker.io v2 true true 0xc0422d8420}]"
time="2017-05-17T13:09:09-07:00" level=debug msg="attempting v1 ping for registry endpoint https://registry-1.docker.io/v1/"
time="2017-05-17T13:09:10-07:00" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
time="2017-05-17T13:09:10-07:00" level=debug msg="Registry version header: '0.8.15'"
time="2017-05-17T13:09:10-07:00" level=debug msg="PingResult.Version: \"0.8.15\""
time="2017-05-17T13:09:10-07:00" level=debug msg="Registry standalone header: ''"
time="2017-05-17T13:09:10-07:00" level=debug msg="PingResult.Standalone: true"
time="2017-05-17T13:09:10-07:00" level=debug msg="Trying to fetch image manifest of stefanscherer/winspector repository from https://registry-1.docker.io v2"
time="2017-05-17T13:09:15-07:00" level=info msg="Image \"stefanscherer/winspector:linux-1.7.1\" is digest sha256:6097f83a1b6f02f504b4a6fbb1f5e582b06b597add10139f7f46f6a4001c8420; size: 1994"
time="2017-05-17T13:09:15-07:00" level=debug msg="Adding manifest references of \"stefanscherer/winspector:linux-1.7.1\" to blob mount requests"
time="2017-05-17T13:09:15-07:00" level=debug msg="Adding manifest \"stefanscherer/winspector\" -> to be pushed to \"stefanscherer/winspectortest\" as a manifest reference"
time="2017-05-17T13:09:15-07:00" level=debug msg="authConfig for docker.io: stefscherer"
time="2017-05-17T13:09:15-07:00" level=debug msg="endpoints: [{false https://registry-1.docker.io v2 true true 0xc0422d9340}]"
time="2017-05-17T13:09:15-07:00" level=debug msg="attempting v1 ping for registry endpoint https://registry-1.docker.io/v1/"
time="2017-05-17T13:09:17-07:00" level=debug msg="Error unmarshalling the _ping PingResult: invalid character '<' looking for beginning of value"
time="2017-05-17T13:09:17-07:00" level=debug msg="Registry version header: '0.8.15'"
time="2017-05-17T13:09:17-07:00" level=debug msg="PingResult.Version: \"0.8.15\""
time="2017-05-17T13:09:17-07:00" level=debug msg="Registry standalone header: ''"
time="2017-05-17T13:09:17-07:00" level=debug msg="PingResult.Standalone: true"
time="2017-05-17T13:09:17-07:00" level=debug msg="Trying to fetch image manifest of stefanscherer/winspector repository from https://registry-1.docker.io v2"
time="2017-05-17T13:09:22-07:00" level=info msg="Image \"stefanscherer/winspector:windows-1.7.1\" is digest sha256:6b2e250422e969702509200254d075c8d1c0b88f09b76d10a33043e3e35bbe41; size: 3656"
time="2017-05-17T13:09:22-07:00" level=debug msg="Adding manifest references of \"stefanscherer/winspector:windows-1.7.1\" to blob mount requests"
time="2017-05-17T13:09:22-07:00" level=debug msg="Adding manifest \"stefanscherer/winspector\" -> to be pushed to \"stefanscherer/winspectortest\" as a manifest reference"
time="2017-05-17T13:09:22-07:00" level=debug msg="Manifest list push url: https://registry-1.docker.io/v2/stefanscherer/winspectortest/manifests/latest"
time="2017-05-17T13:09:22-07:00" level=debug msg="mediaType of manifestList: application/vnd.docker.distribution.manifest.list.v2+json"
time="2017-05-17T13:09:22-07:00" level=debug msg="authConfig for docker.io: stefscherer"
time="2017-05-17T13:09:25-07:00" level=debug msg="Mount of blob sha256:ddbc94156cc3298fd4235bf5d6d775496c4af1a631f8974bd3aa36d954ab81b5 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:ddbc94156cc3298fd4235bf5d6d775496c4af1a631f8974bd3aa36d954ab81b5\""
time="2017-05-17T13:09:27-07:00" level=debug msg="Mount of blob sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93\""
time="2017-05-17T13:09:30-07:00" level=debug msg="Mount of blob sha256:7f8ede2d2484a67ae3a88912d400c46f4f76e8f62f1003ed10c6f95893603781 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:7f8ede2d2484a67ae3a88912d400c46f4f76e8f62f1003ed10c6f95893603781\""
time="2017-05-17T13:09:31-07:00" level=debug msg="Mount of blob sha256:3c752c95ebfb440ae472bd7c28a85adde5844abc783e8e0c12585b3055e9e804 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:3c752c95ebfb440ae472bd7c28a85adde5844abc783e8e0c12585b3055e9e804\""
time="2017-05-17T13:09:33-07:00" level=debug msg="Mount of blob sha256:39c204c948870160e7e0aeaeaf01229492437a04a18345faed5e2733b0d0a64e succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:39c204c948870160e7e0aeaeaf01229492437a04a18345faed5e2733b0d0a64e\""
time="2017-05-17T13:09:35-07:00" level=debug msg="Mount of blob sha256:4d20e2a064f4cbd8241815318cdc2422f2b6ef6384e2af55892a440deca15f72 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:4d20e2a064f4cbd8241815318cdc2422f2b6ef6384e2af55892a440deca15f72\""
time="2017-05-17T13:09:37-07:00" level=debug msg="Mount of blob sha256:f63a734d587e2d53ab4390c2f6e8d586d70eb5995bebf707f395c24d600b8bd5 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:f63a734d587e2d53ab4390c2f6e8d586d70eb5995bebf707f395c24d600b8bd5\""
time="2017-05-17T13:09:38-07:00" level=debug msg="Mount of blob sha256:8b64818b6bdd13c5973a3e8e4a3996a1ff72cecabc29a62e28fa58568d437878 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:8b64818b6bdd13c5973a3e8e4a3996a1ff72cecabc29a62e28fa58568d437878\""
time="2017-05-17T13:09:40-07:00" level=debug msg="Mount of blob sha256:ae59d408cfc3ff127671707ec5580a1b35405e743f8022ca04cd2171cd651316 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:ae59d408cfc3ff127671707ec5580a1b35405e743f8022ca04cd2171cd651316\""
time="2017-05-17T13:09:42-07:00" level=debug msg="Mount of blob sha256:4de619810b10400a3ac26955f47b84d1794f50493f9a24a02d4ffd3b94c7de81 succeeded, location: \"https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/sha256:4de619810b10400a3ac26955f47b84d1794f50493f9a24a02d4ffd3b94c7de81\""
time="2017-05-17T13:09:45-07:00" level=fatal msg="Couldn't mount blobs for cross-repository push: Blob mount failed to url https://registry-1.docker.io/v2/stefanscherer/winspectortest/blobs/uploads/?from=stefanscherer%2Fwinspector&mount=sha256%3Abce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277: HTTP status 202"

The layer with sha256:bce2fbc256ea437a87dadac2f69aabd25bed4f56255549090056c1131fad0277 is the microsoft/nanoserver:10.0.14393.447 base layer, there also is a second sha256:4a8c367fd46d2e2da2a8b0fa02158540e13b3a9015daf9f17d1af354a591492f which is the Windows update layer.

@estesp

This comment has been minimized.

Show comment
Hide comment
@estesp

estesp May 17, 2017

Contributor

Thanks for the extra debug @StefanScherer! So, @clnperez we need to allow 202 HTTP response code for the special case Microsoft base layers. That special case code (properly) notes that those blobs aren't actually created and uses a different response code to handle, and should be allowed here.

I'm going to make the same modification to manifest-tool.

Contributor

estesp commented May 17, 2017

Thanks for the extra debug @StefanScherer! So, @clnperez we need to allow 202 HTTP response code for the special case Microsoft base layers. That special case code (properly) notes that those blobs aren't actually created and uses a different response code to handle, and should be allowed here.

I'm going to make the same modification to manifest-tool.

Add 'docker manifest' command
The workflow will be:

`docker manifest create new-list-ref-name manifest [manifests...]`
`docker manifest annotate new-list-ref-name manifest --os linux --arch arm`
`docker manifest push new-list-ref-name`

- or -

`docker manifest push -f annotated-manifests.yaml`

There is also a `manifest inspect` command to allow for a *shallow pull*
of an image's manifest: `docker manifest inspect manifest-or-manifest_list`.
These by default show a ManifestDescriptor in the case of a single
manifest, or a DeserialedManifestList.

To be more in line with the existing external manifest tool, there is
also a `-v` option for inspect that will show information depending on
what the reference maps to (list or single manifest).

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
@clnperez

This comment has been minimized.

Show comment
Hide comment
@clnperez

clnperez May 17, 2017

Contributor

I pushed fixes for both issues.
🙏 🙏 🙏 @StefanScherer and @estesp for tracking them down.

Contributor

clnperez commented May 17, 2017

I pushed fixes for both issues.
🙏 🙏 🙏 @StefanScherer and @estesp for tracking them down.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 May 22, 2017

Contributor

Note that the CLI has moved repos so this should now go into github.com/docker/cli.

Contributor

cpuguy83 commented May 22, 2017

Note that the CLI has moved repos so this should now go into github.com/docker/cli.

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas May 29, 2017

@clnperez ping us when you have the new PR up 👏

luxas commented May 29, 2017

@clnperez ping us when you have the new PR up 👏

@clnperez clnperez referenced this pull request May 30, 2017

Merged

Add manifest command #138

1 of 1 task complete
@clnperez

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 May 31, 2017

Contributor

Thanks, going to go ahead and close this since it looks like this is all CLI bound code.

Contributor

cpuguy83 commented May 31, 2017

Thanks, going to go ahead and close this since it looks like this is all CLI bound code.

@cpuguy83 cpuguy83 closed this May 31, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment