Unable to connect to registry with client certificate signed by intermediate certificate

[callebjorkell]

We have a docker registry setup where SSL is terminated by haproxy for our private registry. We use a client certificate for authentication. We have been running this setup successfully for quite some time using client certs signed by our root certificate. Recently we started signing client certs with an intermediate certificate instead, and docker doesn't seem to support this setup. We have tried to bundle the intermediate and root certificates in the same .crt file, along with the client.cert, and as two separate .crt files. None of the above allowed us to connect to the registry (see below for error output).

To verify that the server side setup was OK, openssl and a browser was used with the same certs. Both were able to connect. OpenSSL command used:

sudo openssl s_client -debug -connect my.registry.url:443 -CAfile ca.crt -cert client.cert -key client.key

This worked with the intermediate concatenated with the root cert in the same .crt file.

Output of docker version:

Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 22:37:33 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 21:49:11 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.1.19-boot2docker
Operating System: Boot2Docker 1.10.3 (TCL 6.4.1); master : 625117e - Thu Mar 10 22:09:02 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.1 MiB
Name: test
ID: N7P7:5TYU:NGPM:OP3P:L53Y:YPLW:T4TN:TKLR:NR2M:R3JF:ZQEA:KS7G
Debug mode (server): true
 File Descriptors: 10
 Goroutines: 21
 System Time: 2016-03-12T06:28:21.383388796Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

Additional environment details (AWS, VirtualBox, physical, etc.):

docker-machine -> haproxy -> docker registry

Steps to reproduce the issue:

1. Put CA bundle (concatenated root cert and intermediate cert), client cert and key into ca.crt, client.cert and client.key respectively in /etc/docker/certs.d/my.registry.url/
2. Try to pull any image from private repository

Describe the results you received:

Error response from daemon: unable to ping registry endpoint https://my.registry.url/v0/
v2 ping attempt failed with error: Get https://my.registry.url/v2/: remote error: handshake failure
 v1 ping attempt failed with error: Get https://my.registry.url/v1/_ping: remote error: handshake failure

Describe the results you expected:
Being able to pull the docker image from the registry.

Additional information you deem important (e.g. issue happens only occasionally):

[dmcgowan]

@callebjorkell the ca.crt file will not be use for generating the client tls config. Can you try putting any intermediate certificates inside the client.cert and see if that makes a difference. If you are still having issues I will add a test case for this and see if I can reproduce the failure, we have tests for client tls auth but nothing that uses an intermediate. In the end though it is just using https://golang.org/src/crypto/tls/tls.go?s=5758:5829#L183 to generate the certificate information and multiple CERTIFICATE blocks should work just fine in client.cert.

[callebjorkell]

@dmcgowan Thanks for taking a look at this. We have tried the variant you describe as well to no avail. I did try to include that in the description above, but I don't blame you for missing it as I'm not the most coherent person alive 😉

I tried to include the intermediate both before and after the actual client cert, but couldn't get it to work. If you know of any special syntax you need for that (or format), please let me know.

[dmcgowan]

@callebjorkell: I have been able to reproduce an error with an intermediate certificate but not able to root cause it (not sure whether server/nginx configuration, certificate, docker, or golang issue). To help isolate the cause, could you provide the relevant section of your HAProxy configuration. You can track my attempts here https://github.com/dmcgowan/distribution/commits/client-intermediate-test.

[callebjorkell]

I haven't been able to look into this, but we did create a go client for another part of our system recently using the same types of intermediate client cert. Being written in go, we ran into something that might be relevant for this. The tls package does seem to construct a ca bundle, reading all certs in the ca file. However, to get the client cert to use the intermediate, it had to be in the client certificate file, just like you suggested. Having it bundled with the CA didn't work.

At first glance, the setup that you're testing does seem sane to me, I'll try to allocate some time to look at it more closely if needed.

The thing that we tried to verify that the SSL termination was working was using either curl or openssl s_client to connect to the registry with the same certs, showing that the issue was client side.

[thom4parisot]

Same here, we use SSL certificate authentication at the BBC for lots of services instead of passwords.

From what I understand, providing --tlscert and --tlskey only relates to the communication with the docker daemon and thus, this is not intended to be used with the docker login and any operation related to registry communication, like push and pull.

$ curl -I -E "$(id -F)" https://foobar.bbc.co.uk/artifactory/docker/
HTTP/1.1 200 OK

[allingeek]

It been a year, and I still think that registry client certificates are a good idea. Even if the Distribution project doesn't support them I know that many people are terminating TLS with reverse proxies that are capable of client certificate validation.

[dmcgowan]

@allingeek the registry does support client certificates. This issue is specifically for supporting client certificates signed by intermediate certificates without having to put the intermediate certificates into the CA bundle. This issue was not fully root caused so it is unclear whether there is anything that needs to be changed inside of docker or the registry. If you have a go setup successfully using intermediate certificates for client authentication, please share any insight that might help us figure out what needs to be updated. Otherwise it is is possible this is an issue in golang.