Docker Daemon behaves surprisingly and incoherently when a port is specified for the default docker registry

[Alvaro-Santos]

This issue continues from an issue opened today in the docker cli repository. The previous issue can be found here: docker/cli#2372. The previous issue is superseded by this one; it is linked here only to help contextualise this one. The previous issue was opened before I explored the situation further and realised it appeared to be a problem with the docker daemon, not the docker command line interface.

Description

The docker daemon does not correctly handle commands that refer to the docker hub registry (both as index.docker.io and as registry.hub.docker.com) if they explicitly set the port to 443. From my experimentation, I believe this misbehaviour affects the whole of the daemon interface, although my testing was focused on the endpoints for pulling and reading manifests.

This behaviour is extremely surprising as explicitly setting the port works if the registry is not docker hub's (i.e., if the registry is locally hosted), and as accessing the docker hub registry manually (using curl or a web browser) works as expected whether the port is explicitly set or not. Note that the documentation itself explicitly sets ports in some examples: https://docs.docker.com/engine/reference/commandline/pull/#pull-from-a-different-registry#pull-from-a-different-registry. I did not find a single example that set it for index.docker.io --- which makes sense, as it'd be redundant.

This misbehaviour was found in the course of writing a tool that interfaces with registries provided by users as pairs of URLs and ports. Presently, my tool has to special-case the handling of the index.docker.io & 443 combination, which is ugly since it's the default registry (!!).

Steps to reproduce the issue:
Excruciatingly detailed version

Makefile.txt
Results.txt

I have attached 2 files: a Makefile containing rules to run 24 different tests, and a file containing the results I obtained for each of those tests, along with an explanation of each result. The results are in the form [Success/Failure, Expected/Unexpected], with 'Success' meaning that the test manages to pull the image/fetch its manifests and 'Expected' meaning that the success or failure of that test matches the expectations of an user with no a priori knowledge of the tests (i.e., someone who has only read the documentation and is trying out things for themselves).

The Makefile is very verbose, but it is written this way so that it's easy to analyse any given command (I do not expect other users to trust me and run my code without verifying it for myself). The tests were written to run on Ubuntu, with the bash shell and the gnu versions of the utilities used, but I tried to write them as agnostic and portable as was reasonably possible.

Some notes about the Makefile:

• Some of the tests require a local registry to run (a rule exists to create and run the registry). There is no rule to clean up this local registry, you must stop the container and remove it manually;
• Most tests use the alpine:latest image. Most tests also assume that this image is not present, and remove it at the end/the corresponding rule to clean up after the test removes the image;
• Some rules implicitly clean up after themselves immediately. This means they may delete the alpine:latest image;
• Some tests create a temporary file 'docker_io_token.txt' in the Makefile's directory;
• HOST_PORT defines under which host port to bind to the locally-run registry used in some of the tests;
• DOCKER_666_VERSION defines which version of the daemon API the relevant tests will use;
• Instructions on which order the commands should be run can be found in the results file;
• The Makefile rules are as suggestive as I could make them without being overly verbose:
  -- cli: Test uses the docker CLI
  -- lh: localhost registry
  -- port: test sets the port explicitly
  -- clean: cleans up the corresponding rule
  -- np: test does not set the port explicitly
  -- hub-idi: index.docker.io registry
  -- hub-rhdc: registry.hub.docker.com registry
  -- dockpy: Test uses docker-py (version 4.2.0 from PyPi)
  -- curl-666: Test uses curl to connect to the docker daemon
  -- curl-regi: Test uses curl to connect to the docker registry

Some miscellaneous observations I made during these tests:

• docker.py URL-encodes the ':' and '/' in the value of fromImage sent to the /images/create docker daemon endpoint, although this doesn't seem to affect the endpoint's function (tested sending the URL-encoded values too via curl);
• docker.py contacts v1.35 of the docker daemon API, instead of v1.40 as used by docker. Again, this doesn't seem to have any bearing on the success/failure of the tests;
• N.B. for the maintainers: the errors returned by the /images/create endpoint do not seem to match those listed at https://docs.docker.com/engine/api/v1.40/#operation/ImageCreate;
• The docker daemon's logs (enabled as per https://docs.docker.com/config/daemon/#enable-debugging, with a conf file containing only the "debug" key) (which can be viewed with the make rule ubuntu-docker-logs --- written as per https://docs.docker.com/config/daemon/#read-the-logs) list the following when trying to access index.docker.io:443/alpine:latest:

mar 03 20:56:47 penguin-box dockerd[2194]: time="2020-03-03T20:56:47.929879227Z" level=debug msg="hostDir: /etc/docker/certs.d/index.docker.io:443" (which doesn't exist)

This is very strange, as the file doesn't exist (and I don't believe I deleted it) & nothing similar appears in the logs when omitting the port;

• Changing the examples from .../alpine... to .../library/alpine... doesn't seem to make any difference (at least for the tests I tried changing it in);
• The X-Registry-Auth header doesn't seem to do anything for the /images/create endpoint;
• Authentication on the daemon, as per https://docs.docker.com/engine/api/v1.40/#section/Authentication, appears to require both an email and an username. This is strange, as the Docker CLI doesn't require an email. Additionally, the /auth endpoint appears to always return an empty IdentityToken. This has been reported in other issues (Docker API 1.37 returning empty IdentityToken #38830 (comment)), but as it may be surprising to prospective assignees to this issue, I thought to mention it here.

And some conjectures that might help the assigned maintainers:

• Trying to access registry.hub.docker.com through the docker CLI, docker-py, or by curling the daemon always seems to fail. This may be because the daemon is trying to access the auth endpoint as it would for index.docker.io (-> auth.docker.io/token) when accessing registry.hub.docker.com (-> auth.registry.hub.docker.com/token --- or perhaps with elisions, like auth.hub.docker.com/token), and failing to do so as those subdomains don't exist, instead of correctly accessing auth.docker.io/token as indicated by the WWW-Authenticate header even when accessing registry.hub.docker.com;
• The conjecture above doesn't explain why the weird behaviour herein described (failing to access the default registry when a port is specified) also happens for index.docker.io:443 though, and why it doesn't happen when accessing a local registry like localhost:3311. However, note that the locally hosted registry used in these examples is insecure, so there is no need to auth.;
• Also, note that auth.docker.com/token returns a 503, unlike the other speculatory auth URLs listed above. This error shows up in many of these tests, lending some credence to the hypothesis that the daemon is being "stupid" and ignoring the value of the WWW-Authenticate header, trying instead to build the URL to the auth endpoint manually.

Short version

1. Run docker pull index.docker.io/alpine:latest and verify that it works.
2. Delete the pulled image with docker image remove index.docker.io/alpine:latest.
3. Run docker pull index.docker.io:443/alpine:latest and verify that it fails spectacularly.

Describe the results you received:

The command in step 3. of the Short version above fails.

Describe the results you expected:

I expected commands 1. and 3. of the Short version above to both produce the same result (pull the image), as port 443 is (what I expect, but did not confirm by intercepting the network traffic, although it is supported both by common sense and comments on older issues in this repository: #40355 (comment)) the port that the daemon will default to contacting anyway.

Additional information you deem important (e.g. issue happens only occasionally):

This behaviour happens consistently, it is not transient (tested intermittently over a period of ~14 hours). This behaviour also does not depend on logged in status, nor does it appear to depend on previously issued commands or on how any docker component is configured. I have also confirmed that the docker hub registry is available, and my internet connectivity has been stable over the 14 hours and tens (hundreds?) of tests that I ran.

The behaviour seems to be attributable to the docker daemon, as it manifests when using the docker CLI, the docker python SDK (docker-py), and when CURLing to the docker daemon, but not when CURLing directly to the docker hub (or a local) registry.

Please also refer to the issue linked at the top of this text box, and to the Excruciatingly detailed version above.

Output of docker version:

Client: Docker Engine - Community
Version: 19.03.6
API version: 1.40
Go version: go1.12.16
Git commit: 369ce74a3c
Built: Thu Feb 13 01:27:49 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.6
API version: 1.40 (minimum version 1.12)
Go version: go1.12.16
Git commit: 369ce74a3c
Built: Thu Feb 13 01:26:21 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.12
GitCommit: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

Output of docker info:

Client:
Debug Mode: false

Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 19.03.6
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 7
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-88-generic
Operating System: Ubuntu 18.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.661GiB
Name: penguin-box
ID: M54W:2ZHZ:46WS:COAQ:WH4K:26Y3:44C2:2KBG:T75H:K46C:7AGE:AXJT
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 27
Goroutines: 39
System Time: 2020-03-04T02:20:04.768077102Z
EventsListeners: 0
Username: malloc1024
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

Additional environment details (AWS, VirtualBox, physical, etc.):
N/A

Comments would be appreciated.