Docker pull with "allow-nondistributable-artifacts" still downloads foreign layer from foreign server

[rodrigozr]

Description

Pulling an image (from a local private registry) that depends on a foreign layer still downloads the layer from the foreign server, even when "allow-nondistributable-artifacts" is correctly configured and the foreign layer is present in the private registry.

Steps to reproduce the issue:

1. docker pull microsoft/windowsservercore:10.0.14393.1198
2. Configure "allow-nondistributable-artifacts": ["myregistry.local"] in the daemon and restart
3. docker tag microsoft/windowsservercore:10.0.14393.1198 myregistry.local/microsoft/windowsservercore:10.0.14393.1198
4. docker push myregistry.local/microsoft/windowsservercore:10.0.14393.1198
5. Delete all local images
6. docker pull myregistry.local/microsoft/windowsservercore:10.0.14393.1198 (GOOD - Downloads from local registry)
7. Create a new image with "FROM windowsservercore:10.0.14393.1198" and push it to "myregistry.local"
8. Delete all local images
9. Pull the image created on step 7 (BAD - Downloads foreign layer from MS servers)

Describe the results you received:
The foreign layer is downloaded from MS servers

Describe the results you expected:
The foreign layer should be downloaded from "myregistry.local"

Output of docker version:

Client:
 Version:      17.06.1-ce-rc1
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   77b4dce
 Built:        Fri Jul 14 07:36:58 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.1-ce-rc1
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   77b4dce
 Built:        Fri Jul 14 07:45:51 2017
 OS/Arch:      windows/amd64
 Experimental: true

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.06.1-ce-rc1
Storage Driver: windowsfilter
 Windows:
Logging Driver: json-file
Plugins:
 Volume: local
 Network: l2bridge l2tunnel nat null overlay transparent
 Log: awslogs etwlogs fluentd json-file logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 14393 (14393.1480.amd64fre.rs1_release.170706-2004)
Operating System: Windows Server 2016 Standard
OSType: windows
Architecture: x86_64
CPUs: 8
Total Memory: 63.96GiB
Name: US91760RHS02
ID: CJVJ:DSWX:OQ2H:EYWP:QBFC:Q4F5:WSZG:B3OW:4VSO:2MNS:ZG6I:VUCG
Docker Root Dir: C:\ProgramData\docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: -1
 Goroutines: 19
 System Time: 2017-07-21T12:37:43.8075815-05:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Tested on AWS and physical machines, on both Windows 10 and Windows Server 2016

[thaJeztah]

ping @jstarks @stevvooe @friism

[rodrigozr]

Side note: this was reproduced also using the "Azure Container Registry" as the backend for the private registry.

[sixeyed]

@rodrigozr have you tried with any other registry? I have tested this on a local registry running in a Windows Docker container, and the workflow is as expected - the foreign layers are pushed to the local registry and pulled from there on other clients.

[rodrigozr]

@sixeyed, we have tested with AWS ECR and Azure CR.
What is the registry image that you are using? I thought the registry only had Linux images available. Thanks

[sixeyed]

@rodrigozr yes, the official image only has Windows variants right now, I'm using a custom one: sixeyed/registry.

[friism]

@rodrigozr what's your use case? Please note that allow-nondistributable-artifacts is designed only for specialized uses such as deploying dockerized apps onto air-gapped systems that don't have internet access.

To get the behavior you're looking for your should use FROM myregistry.local/microsoft/windowsservercore:10.0.14393.1198

[rodrigozr]

@friism, the use case is similar to air-gapped systems. Some locations we will deploy to may/will not have access to Microsoft servers - only to our own servers.

If your suggestion of referencing the local image at build time is the only solution, it will probably not work for us, as the registry address will change depending on the environment (development, QA, staging, production). Shouldn't the layer ID / hash be the most important unique identifier for it, rather than the registry where it was located at build time?

Thanks in advance for any assistance provided!

[friism]

Shouldn't the layer ID / hash be the most important unique identifier for it, rather than the registry where it was located at build time?

microsoft/windowsservercore:10.0.14393.1198 is an implicit reference to the global namespace on Docker Hub/Store I'm afraid.

referencing the local image at build time is the only solution, it will probably not work for us, as the registry address will change depending on the environment (development, QA, staging, production)

You might be able to template that and re-build for each environment: https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact

It'll probably introduce some other problems, but you could also look at simply pre-loading the base images that you care about on the production systems where you'll be running containers. Eg. with docker save and docker load. When pulling an image based on those base layers Docker will realize they're already present and not attempt to pull

[rodrigozr]

Thanks for the information @friism. But, on that case, why would it work with a local private registry, as mentioned by @sixeyed? After his findings, I was assuming that this was actually an issue with Azure CR and AWS ECR, and not an issue with the client itself.

[sixeyed]

My workflow was to re-tag the Store image and include my registry domain - so:

docker image pull microsoft/windowsservercore
docker image tag microsoft/windowsservercore registry.sixeyed:5000/microsoft/windowsservercore
docker image push registry.sixeyed:5000/microsoft/windowsservercore

Then the foreign layers are in the local registry, but you still need to use the domain prefix to reference the image from that registry instead of Store.

[friism]

@rodrigozr yeah, so the reason this works for @sixeyed is that he uses the fully qualified image name including the hostname of the registry in the FROM instruction

[rodrigozr]

Thanks for the clarifications @friism and @sixeyed .
Let me change the approach a little and see if you guys can give some guidance on how to achieve what I need to achieve with Docker:

1. Deployment of images will happen to potentially 36.000 physical locations around the globe
2. Some of those locations are not allowed to access public CDNs or servers, so all image layers must come from our own servers
3. For the sake of clarity: All those locations are part of our corporation (we are not trying to distribute foreign layers to any third-party or similar, as that would most likely not be allowed)
4. Different private registries may be necessary per geographical location. For example, China and Russia-based locations will most likely require us to place the private registry into our on-premisses private cloud infrastructure. This means that we will not know at image build time the name for the private registry that will be used in production
5. Different registries may also be needed during the software development lifecycle phases (DEV -> QA -> Staging, etc). But we must ensure that the images being promoted between those environments are exactly the same (I.e.: Their hash / ID is exactly the same in QA registry as it was in DEV registry).

From the documentation of "allow-nondistributable-artifacts", it sounds like it was designed to solve similar scenarios:

This option is useful when pushing images containing nondistributable
artifacts to a registry on an air-gapped network so hosts on that network can
pull the images without connecting to another server.

(Emphasis is mine)

Any guidance will be greatly appreciated! Thanks again

[friism]

I'd separate out dev/staging/etc from production. Hopefully you don't have the non-foreign layer requirement before you have to deploy to production?

For production, and with the way that Docker currently works with foreign layers, I'm not sure the Docker registry infrastructure and push/pull is going to work out for you. I would simply docker save the images and make those tarballs available at production locations some other way (eg. SMB share or whatever floats your boat) and then docker load them as needed.

[rodrigozr]

On that case, would there be a way to save only one of the layers (from the top of the stack) when doing that?
The base server image is composed of two layers today (maybe more in the future)... I certainly don't want to push a ~7GB file to 36K locations when only the ~2GB layer was modified. But that later is at the top of the stack, so there is no way to save it by itself AFAIK.
Doing a docker pull would download only the modified layer, and that would be crucial to the success of this project.

[friism]

I think it would work if you just docker save and docker load the microsoft/windowsservercore:<tag> base images in all the locations. And then don't try to do anything clever with allow-nondistributable-artifacts. The main problem is that you have to be very disciplined about only building and deploying images based on the windowsservercore base layers that you know are pre-deployed in production locations. That'll be difficult as Microsoft ships updates to those fairly frequently.

[rodrigozr]

Okay, so it looks like the documentation for "allow-nondistributable-artifacts" should be updated, as it does not solve the use-case of air-gapped systems.

Please notice that it is not just "defaulting" to the foreign servers... When running a docker pull in an air-gapped system it literally fails to download the foreign layers after some retries.

I'm a bit confused now... Can you please elaborate what use-case that newly-introduced configuration option is addressing?

[friism]

Can you please elaborate what use-case that newly-introduced configuration option is addressing?

1. Docker image is built outside air-gap
2. Image is transported over airgap
3. image is pushed to private registry running in airgapped datacenter
4. Docker hosts in airgapped datacenter can pull and run said image

Below is an example flow using multiple Docker engines running on my laptop to simulate. They use separate graph-driver directories, so they behave as separate hosts for our purposes.

Start a local private registry

This simulates registry running behind airgap. We run in its own daemon.

.\dockerd.exe -H "npipe:////./pipe//docker_registry" --data-root c:\registry-daemon

Now start the registry on that daemon:

docker -H "npipe:////./pipe//docker_registry" run -d --ip 172.18.248.236 -v c:\registry-data:c:\dat
a -p 5000:5000 sixeyed/registry
Unable to find image 'sixeyed/registry:latest' locally
latest: Pulling from sixeyed/registry
bce2fbc256ea: Pull complete
3ac17e2e6106: Pull complete
dd559b56cddd: Pull complete
936a52bc4b03: Pull complete
554ff58a1dab: Pull complete
7b5527e79ea2: Pull complete
5a2c776caee1: Pull complete
b19db2c02c3e: Pull complete
f11fda1ad1a8: Pull complete
Digest: sha256:39b831ac73e60c510f336a64e5efe20abb2305d0ea58ba76657e9b4dab0f070b
Status: Downloaded newer image for sixeyed/registry:latest
f05592988fb42b014c88dff7ae26ecfb0bf5ec5665f956d06ce426ea1ee07022

Tag and push image to local registry

For this example I'm just tagging a random Windows-based image on my machine. Note that I have to configure my daemon to be able to push. This step simulates someone building an image, transporting it physically over the airgap and pushing to airgapped registry:

{
  "allow-nondistributable-artifacts": [
    "localhost:5000",
    "172.18.248.236:5000"
  ],
  "registry-mirrors": [],
  "insecure-registries": [
    "localhost:5000",
    "172.18.248.236:5000"
  ],
  "debug": true,
  "experimental": true
}

docker tag broyal/samplefxapp-api:1.0 172.18.248.236:5000/myimage
docker push 172.18.248.236:5000/myimage
The push refers to a repository [172.18.248.236:5000/myimage]
78ab2f301869: Pushed
f819641c91e0: Pushed
525dfaaf4ca3: Pushed
bbbf161599ff: Pushed
bfdfe56563ee: Pushed
efc8818e7cf3: Pushed
ce74595d58f5: Pushed
5b4aace84103: Pushed
1f2f3eb32edc: Pushed
0451551dda21: Pushed
c28d44287ce5: Pushed
f358be10862c: Pushed
latest: digest: sha256:e7089d948f3e7a9556f33d5ce4ae42302b3f2f61687e9cf310589afe305d9712 size: 2844

Use image from airgapped machine

At this point I turned off internet access on my machine and started a third daemon, representing an airgapped host that can consume images from the private, airgapped registry:

.\dockerd.exe -H "npipe:////./pipe//docker_3rd" --data-root c:\3rd-daemon

docker -H "npipe:////./pipe//docker_3rd" pull 172.18.248.236:5000/myimage
Using default tag: latest
latest: Pulling from myimage
6c4061ac9d96: Extracting [=============>                                     ]  1.099GB/4.175GB
d6752e92e2a1: Download complete
90591b4a81ed: Download complete
9f7b4d854452: Download complete
f28cc477da07: Download complete
78efd1bcc18d: Download complete
2717bc5bc1a2: Download complete
490eeda55971: Download complete
f57481977836: Download complete
7aa90d27f0eb: Download complete
2ce4053a11c9: Download complete
5d36d2dec087: Download complete

Summary

So in your case, for each of the "air-gapped" places you want to make the images available, you will have to transport the image there one way or another, then docker tag <your-image> <fully-qualified-hostname-of-airgapped-registry>/yourimage and then docker push <fully-qualified-hostname-of-airgapped-registry>/yourimage. Then other Docker hosts in that air-gapped deployment will be able to consume the image with docker pull <fully-qualified-hostname-of-airgapped-registry>/yourimage.

[rodrigozr]

Awesome explanation @friism!

In fact, that is exactly how I need it to work, and is somewhat similar to my original step by step report.

You didn't have to rebuild broyal/samplefxapp-api:1.0" with a different FROM instruction, which was a big concern from my part. That is great.

Now we probably just need to find out why I'm seeing different results...

The main differences that I can see between my tests and yours are:

1. I didn't use multiple local engines, but that should not affect anything. I used multiple machines.
2. I'm using hosted private registries. (Azure CR & AWS ECR). If this feature relies on registry changes, that could explain the different results.
3. The image that I'm pulling (broyal/samplefxapp-api:1.0 on your case) was pushed to the private registry by our CI servers, which don't have the configuration for "allow-nondistributable-artifacts". If there is any metadata modification happening at the time of the push based on that config, that might explain it. I'm personally hoping that this is the cause, as it would be the simplest to solve.

I will retest taking those factors in consideration!

Thanks again

[friism]

pushed to the private registry by our CI servers, which don't have the configuration for "allow-nondistributable-artifacts"

That's the problem.

[rodrigozr]

Thanks @sixeyed and @friism for all the support! It is working fine now 👍

Amazingly enough, all I had to do was to run this command:

• docker pull myregistry.local/myimage && docker push myregistry.local/myimage

As my laptop has the configuration for allow-nondistributable-artifacts, this seems to rewrite the image metadata in the server to point to the correct location, and now the docker pull on any other machine downloads the foreign layers from the private registry.

The long-term solution will be to configure our build agents with allow-nondistributable-artifacts.

Closing this issue!

[sixeyed]

Glad you have it working @rodrigozr. One thing may help with multiple registries. The domain part of the image name just uses DNS when Docker resolves the registry server. So you could have your image called registry.internal/my-repo/my-image and refer to it the same way from every host, but use DNS to point registry.internal to different registry servers.

[rodrigozr]

Thanks for the suggestion @sixeyed. But wouldn't that cause the HTTPS certificate to be considered invalid?

[sixeyed]

Good point, yes. In my local environment I'm just using HTTP :)