Fetch docker images without docker command. e.g. with wget

[LinuxEngr]

I need to set up a Docker repository on a private network without Internet access.

I need to fetch files with wget/curl and then transfer them via a disk to our private network.

I can't do something like docker pull to populate our local registry.

So, 1. I have found a variety of documentation about setting up a local registry but none of it is detailed.
2. I have found nothing about fetching selected images with, say, wget.

thanks.

[RichardScothern]

You cannot use docker pull to populate a registry. The registry stores content via docker push.

1. What is your use case that the documentation is not detailed enough to fulfill?
2. The registry has a REST API which you can use to get, get and put docker content. This is documented here

[LinuxEngr]

Thanks Richard but I don't see how that link helps at all. That appears to be another example of documentation that doesn't address my "use case".

The linked documentation does discuss pulling images and the apparent need to first fetch a manifest but 1. there are no examples; 2. there are no hostnames/URL's of from where to fetch; ...

From what I have read about mirroring a registry it is always described as a way to cache images that are pulled. I don't have that ability. I would like to fetch images with something like wget or curl and build the registry. I can't run a docker command with Internet connectivity to get images.

Is there no documentation on how to mirror a Docker repository with something like wget????

[RichardScothern]

I'm uncertain of where the confusion lies. You have stated that you wish to run a private registry and populate it with HTTP. I provided you details of the registry API which describes how to interact with a registry over HTTP. The API lists endpoints (which are URLs) for you to call. The hostname is whatever host and port you run your private registry on.

The reason that this is not documented is because it is a highly unusual use-case. You would essentially have to implement the registry side code of a docker engine.

Your other option is to run a private registry with a filesystem driver on a machine that is on a different network. Populate it via docker push, tar the storage directory onto a disk and untar it into the storage area on the machine hosting your private registry. This is not future-proof (we don't guarantee the on disk layout between versions) but
will solve your problem.

[LinuxEngr]

Thanks Richard, but I don't think we are communicating.

Firstly, I am not trying to populate a registry with HTTP. I want to get the images and re-create a registry on a network that is not connected to the Internet. Having to populate a private registry with a docker push doesn't seem to make sense. I don't have anything to push. I wanted to be able to download images via wget/curl , perhaps if push is the only way that one creates a local registry then I would do that step.

Also, I am not worried about future proof. So if there were, for example, an existing tar/cpio/iso/... image to download to "prime" a local registry that would work.

As for highly unusual, I would think folks having secure networks unconnected to the Internet would not be so highly unusual. I guess you just haven't heard about folks in that situation wanting to use docker internally yet.

Also, if the only way to fetch Docker images is to fetch them via Docker commands instead of wget/curl then I have to set up some dummy docker machine and tar up its repository. Seems like just fetching images with wget/curl would make more sense and be a lot faster.

I would think, for example, that you are rsyncing or using some other means to mirror your repository. Can't others do that too?

thanks.

[RichardScothern]

@LinuxEngr : I'm going to close this for now. If you have questions about distribution, then you can reach the community with them on the docker-distribution IRC channel.

[viperblack]

@linuxeng: Are you need build a local repository for developers or others can pull checked or authorizated images? You can populate your local repository using tar images like this.

[gcs278]

This is an issue for me. Our servers are not connected to the internet and I am unable to install docker on any other device that is connected to the internet. Being able to wget a docker container would be great.

[jonathankochems]

This is an issue for me too. I need to use docker in a private network. External content can only be downloaded through a process that is essentially doing a wget (the process performs various cybersecurity checks). I do not have any ability to change/extend/improve this process, but would benefit hugely if docker images could be downloaded via wget and then used to populate a registry inside the private network.

[erikbgithub]

@gcs278 @jonathankochems Not sure why you two ignore @viperblack's response but this contains the solution. A stackoverflow answer here: http://stackoverflow.com/questions/23935141/how-to-copy-docker-images-from-one-host-to-another-without-via-repository

[jonathankochems]

@erikbgithub (@gcs278 @viperblack @LinuxEngr). Unfortunately this is not a solution to my usecase. The cybersecurity process that external content has to go through would reject any tarball that I exported outside of the network and hosted separately. I would only be able to instruct the download as a wget from the "official" source, hence my +1 above. Other protocols are blocked and the process is outside of my control.

[cniweb]

In our organization, direct Internet access is not possible. For us, it would be good to download docker images directly as a TAR. +1 for a direct download link in docker hub

[erikbgithub]

Yeah, I definitely won't argue against that @jonathankochems. Alright, I just didn't get your context. I also like the direct download.

It should already be possible but would probably require a frew previous curl requests to gather the corresponding data. I'm confident that it could be scripted.

[cniweb]

But we only have the possibility to download data using a browser (as a Citrix session) (HTTP, HTTPS or FTP).

[joshland96]

I am dealing with the same situation @cniweb. Is there a way to send security certificates through the CLI when Docker attempts to access files via http? A browser seems to do this automatically.

[cniweb]

We have only access to files via browser!

[joshland96]

@cniweb I understand, I'm in the same boat. I'm asking the broader question if anyone has determined a way to extend those http access privileges beyond browsers, particularly to the command line/terminal.

[amandadebler]

Here's what I do in PowerShell to get stuff from PowerShell Gallery from behind an NTLMv2 authenticated proxy - combining that technique with the API info Richard linked to earlier might work out for Windows clients. Bonus: does NOT require putting your AD account credentials anywhere!

https://gist.github.com/amandadebler/ecb3f6ea79ec939acdf12e25710040e1

(putting this here now so that I'll remember the conversation, and in case inspiration strikes someone else to create BootlegDockerRegistryClient first)

Other thing that I think I got to work before, but absolutely doesn't work with Kubernetes: MITMing yourself with Fiddler, which also re-uses IE's authentication.

[Klaas-]

In case anyone else stumbles over this - I think a solution is something like this: https://github.com/moby/moby/blob/master/contrib/download-frozen-image-v2.sh

[landswellsong]

@Klaas- I officially owe you a beer

[yangxinyun]

@Klaas- Hi Klaas, do youknow any windows platform counterpart for this solution?

Thanks in advance

[amandadebler]

@yangxinyun the start of a solution by @so0k: https://gist.github.com/so0k/b59382ea7fd959cf7040

[jimongit]

to add up @Klaas- there is even a python solution https://github.com/NotGlop/docker-drag/ very simple one

[gbenmansour]

@Klaas- do you know if there is something similar for docker push. I'm struggling with a an issue, and want to debug docker push http requests. Thanks in advance.

[bofei222]

@Klaas- Hi Klaas, do youknow any windows platform counterpart for this solution?

[tp1050]

The reason that this is not documented is because it is a highly unusual use-case. You would essentially have to implement the registry side code of a docker engine.

May be I am very late to the this conversation, buta a)what is wrong with an UNSUAL usecase to anything? arent we doing computer scicene ? the whole computer feild is an unsual usecase of electronics!!! it bugs me when I read anything like anywhere on any forum!!! Simple put the Docker command is excuting some form of FETCH mechanism and it shoudl fairly possible to recreate that! Given that almost everything that is used in creating Docker et al comes from people who worked their ass off for free for ages to create the freebees that lead to this, I do not see why anyone should even question an Unsual UseCase.

Secondly or b) I have hit this issue many times when I longer have acess to internet but I have the image, if there was a way to inject the image inside the local registry or better yet just copy it in the right place then it would be early Xmas to me.