x509: certificate is not valid for any names, but ...

[ghost]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

docker login https://our.company.registry:5000 with self signed server certificate will succeed.

Actual behavior

Got error message:

Error response from daemon: Get https://our.company.registry:5000/v2/: x509: certificate is not valid for any names, but wanted to match our.company.registry

I checked the CA and server certs several times with multiple tools. Everything is valid and contents look fine. CN is properly set. I have also written a small GO script, making a TLS connection to the given host with no such error message.

Trying curl https://our.company.registry:5000/v2/ --cacert /etc/docker/certs.d/our.company.registry:5000/ca.crt also works well:

* About to connect() to our.company.registry port 5000 (#0)
*   Trying 1.2.3.4...
* Connected to our.company.registry (1.2.3.4) port 5000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/docker/certs.d/our.company.registry:5000/ca.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=our.company.registry
*       start date: Feb 12 10:46:00 2018 GMT
*       expire date: Feb 12 10:46:00 2019 GMT
*       common name: our.company.registry
*       issuer: CN=DevOps CA,OU=DevOps,O=Company,L=Location,ST=State,C=DE

Turning on logging and debug for docker engine gave no more helpful information.

Digging into the GO x509 source code reveals no hint, why there shouldn't be any names in the server certificate.

Steps to reproduce the behavior

According to https://docs.docker.com/engine/security/certificates/

Add CA certificate to /etc/docker/certs.d/our.company.registry:5000/ca.crt
Execute docker login with username and password.

Output of docker version:

Client:
 Version:       17.12.1-ce
 API version:   1.35
 Go version:    go1.9.4
 Git commit:    7390fc6
 Built: Tue Feb 27 22:15:20 2018
 OS/Arch:       linux/amd64

Server:
 Engine:
  Version:      17.12.1-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   7390fc6
  Built:        Tue Feb 27 22:17:54 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-693.11.6.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.796GiB
Name: sandbox
ID: O6VD:6OPJ:4IDH:IX6V:HF6W:OEYH:XV2P:53BS:2CIW:U3N3:FZBR:ZGIZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 32
 System Time: 2018-03-05T12:36:36.766877949Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.)

CentOS 7 with all updates.
Linux sandbox 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Running on VirtualBox 5.2 under Windows 7

[thaJeztah]

/cc @n4ss @cyli

[cyli]

Could the registry be redirecting to an auth server, for which that certificate may not be valid? For example: https://docs.docker.com/registry/spec/auth/token/#requesting-a-token. Is that a possibility, or is it just htpasswd?

[ghost]

There is no redirect and only Basic Auth.

I can reproduce this on a plain Centos 7 installation without modifications. Attached the Vagrant file and some bootstrap script to recreate the VM.

Vagrantfile

Vagrant.configure("2") do |config|
	config.vm.box = "centos/7"
  	config.vm.provision "shell", path: "bootstrap.sh"	
end

bootstrap.sh

yum update -y
yum upgrade -y

yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
    
yum install -y docker-ce

systemctl daemon-reload
systemctl enable docker
systemctl start docker

The error occurs even without the CA certificate, means before the certificate chain is validated.

[ghost]

Did some further research and changed the bootstrap.sh to not install the latest docker-ce version, but 17.09.

Now everything works as expected. (after manually add the CA certificate)

docker login our.company.registry:5000
Username: foo
Password: baar
Login Succeeded

docker version

Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:23:40 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:25:03 2017
 OS/Arch:      linux/amd64
 Experimental: false

Where is the difference between docker-ce 17.12 and 17.09? The GO version (1.8.3 vs. 1.9.4)?

[cyli]

@gibma Thanks for trying all of this out and providing the vagrant info! I tested your vagrant setup against a registry I spun up with a self-signed CA-signed server cert, and I am not getting the same error, so we may have to just dig into the code some more to see where else it may be making connections. I just wanted to comment so you know you are not being ignored.

[ghost]

@cyli Thank you for taking care of it. I do not want to rule out that our certificate is the cause, but I have no idea how to figure this out. Unfortunately, I can not provide it for privacy reasons, and our registry is not accessible from the Internet anyway.

If I can help in any way to find the cause, let me know

[ghost]

Argh... Found it. The problem is not with Docker, but with golang. I will create an issue there.

Our certificate has the following attributes:

Subject:
    CommonName: our.company.registry
SubjectAlternateNamens:
    IP-Address: 1.2.3.4
    IP-Address: 1.2.3.5

(http://www.alvestrand.no/objectid/2.5.29.17.html)
I don't know if this is a valid combination, but every other tool says this certificate is valid.

So when docker (17.12) is verifying the certificate, the following golang (1.9) code is executed: (https://github.com/golang/go/blob/release-branch.go1.9/src/crypto/x509/verify.go)

...
90   if c.hasSANExtension() {
91       valid = strings.Join(c.DNSNames, ", ")
92   } else {
93       valid = c.Subject.CommonName
94   }
...
97   if len(valid) == 0 {
98       return "x509: certificate is not valid for any names, but wanted to match " + h.Host
99   }
...

It checks if the extension is present (it is), and joins the alternate DNSNames (which we don't have, only IP-Addresses). So after this, valid is empty an the error is thrown. 👎

These lines of code have actually changed between golang 1.8 and golang 1.9. In golang 1.8, the following was executed:
(https://github.com/golang/go/blob/release-branch.go1.8/src/crypto/x509/verify.go)

...
90   if len(c.DNSNames) > 0 {
91       valid = strings.Join(c.DNSNames, ", ")
92   } else {
93       valid = c.Subject.CommonName
94   }
...
97   if len(valid) == 0 {
98       return "x509: certificate is not valid for any names, but wanted to match " + h.Host
99   }
...

Here, the if clause does not match, and the c.Subject.CommonName is used. (and is valid). 👍

Therefore this issue can be closed. Thanks for your support!

[cyli]

Thanks so much for digging into this @gibma! I'll close the issue then.