x509: certificate signed by unknown authority

[susandiamond]

I am using docker registry 2.4 and docker engine 1.10.3 on Ubuntu 14.04. I created a self signed certificate following the instruction in docker community. But I kept getting the "x509: certificate signed by unknown authority" error.
It is a self signed certification, why do I get the error instead of a warning? Docker push and pull failed because of this.

[dmcgowan]

Are you able to curl https://dockerrepo-v2-03.wdc.dev.ibmcsf.net:5000/v2/ from the machine you are pulling from?
Also it is not recommended to add the CA to both /etc/docker/certs.d and your system chain. Testing with your system chain and curl will be a good indication of whether the certificates are correct and being used by the registry.

[susandiamond]

I am able to do:
root@i-5553:/home/cloudadmin# curl -k -X GET https://xxxxx/v2/wdcloud/wdcjava/tags/list
{"name":"wdcloud/wdcjava","tags":["latest","test"]}

I removed the certs in /etc/docker/certs.d, but it still does not work.

I am surprised to find out that I still able to pull image from xxxxxxxx if I configure my client side docker deamon with DOCKER_OPTS="--insecure-registry 10.0.0.0/8" to /etc/default/docker. Docker registry xxxxxxxxx is running in secure mode. I should not able to pull image from it in unsecure mode, right? is this a defect?

[dmcgowan]

Insecure mode will still attempt to use TLS, it will just allow certificates from unknown authorities. Can you send your docker daemon logs and also the link to the docs you used for generating the certificate.

[susandiamond]

The link to the docs I used for generating the certificate:
https://github.com/docker/distribution/blob/master/docs/insecure.md
The Using self-signed certificates section

Attaching the docker daemon log and the docker registry container log

[susandiamond]

any updates?

[dmcgowan]

@susandiamond I tried your steps and the steps in the provided documentation and was not able to replicate your issue.

Here is my log of testing it out. If you can attempt to do something similar in your environment maybe we can see if this is environment or version issue. Note a step I did not show was to update my /etc/hosts to add an entry for "local-registry" to a non-127.0.0.1 address.

$ mkdir certs
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/dockerrepo.key -x509 -days 365 -out certs/dockerrepo.crt -subj /CN=local-registry
Generating a 4096 bit RSA private key
...............................................++
............................................................................................++
writing new private key to 'certs/dockerrepo.key'
-----
$ docker run -d -p 1731:5000 \
-e REGISTRY_STORAGE_DELETE_ENABLED=true \
-e REGISTRY_STORAGE_REDIRECT_DISABLE=true \
-v `pwd`/certs:/certs \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/dockerrepo.crt \
-e REGISTRY_HTTP_TLS_KEY=/certs/dockerrepo.key \
registry:2.4
a8b928b4758b67123c3e9941ea014f6f6beb3a692d8bd75074aa349622787f5e
$ docker push local-registry:1731/alpine
The push refers to a repository [local-registry:1731/alpine]
Get https://local-registry:1731/v1/_ping: x509: certificate signed by unknown authority
$ sudo mkdir /etc/docker/certs.d/local-registry:1731/
$ sudo cp certs/dockerrepo.crt /etc/docker/certs.d/local-registry:1731/ca.crt
$ docker push local-registry:1731/alpine                                     
The push refers to a repository [local-registry:1731/alpine]
8f01a53880b9: Pushed 
latest: digest: sha256:6285471715d8c331a3ac3cc329fe3f54e91b62dee05af3eb7c90e899e37929e0 size: 528
$ docker rm -f a8b928b4758b67123c3e9941ea014f6f6beb3a692d8bd75074aa349622787f5e
a8b928b4758b67123c3e9941ea014f6f6beb3a692d8bd75074aa349622787f5e

[RichardScothern]

@susandiamond did you have any luck with your setup?

[susandiamond]

@RichardScothern @dmcgowan: thanks for the details. I am able to make it work. My docker client is able to push and pull docker images to a secure docker registry now. I have another open issue that I am not able to make it work with use HAProxy as a frontend load balancer to pass through the traffic to Docker registries after enabling SSL in the docker registries. Could you shed some lights on how HAProxy should be configured to do so?

[dmcgowan]

@susandiamond we have many recipes for setting up the registry with different load balancer configurations but unfortunately none with HAProxy. https://docs.docker.com/registry/recipes/. If you are going to terminate SSL on the registry and not within HAProxy then you will need to setup your load balancer as a TCP proxy, otherwise setup HAProxy to do the termination. I am glad you go your certificates working.

[susandiamond]

@dmcgowan I am able to get it SSL work with HAProxy and terminate SSL on the registries. But I still can pull image from the registries directly if I have DOCKER_OPTS="--insecure-registry 10.0.0.0/8" set to /etc/default/docker at the client docker daemon. Is there a way to disable the insecure port at the docker registry so we only support secure calls?

[dmcgowan]

@susandiamond Don't add the insecure registry flag to registries you don't want to contact over HTTP. To best protect your registry it should be on a network only reachable by haproxy, and if you don't want your engine to be able to reach it then it should not be in that protected network. If you cannot secure the network your registry is running on then I would suggest only accept SSL connections by the registry.

[susandiamond]

@dmcgowan I didn't add insecure registry to my docker registries. I think it is enable by default in the docker image of the docker registry v2.4. Following is the configuration I used to start the docker registy:
$ docker run -d -p 1731:5000
-e REGISTRY_STORAGE_DELETE_ENABLED=true
-e REGISTRY_STORAGE_REDIRECT_DISABLE=true
-v pwd/certs:/certs
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/dockerrepo.crt
-e REGISTRY_HTTP_TLS_KEY=/certs/dockerrepo.key
registry:2.4

Is there a flag to explicitly disable the insecure connection?

[dmcgowan]

The flag I was mentioning is on the engine --insecure-registry. Disabling insecure connections on the registry IS enabling SSL. If you have questions about setup please reach out on IRC "#docker-distribution", closing this since the original question has been resolved.

[dminca]

Thanks to our friend for providing us a solution @andrei1489 , you can make it work without using --insecure-registry:

➜  Repos/work  mkdir -p /etc/docker/certs.d/docker-registry.foodomain.org                                         
mkdir: cannot create directory ‘/etc/docker’: Permission denied                                                  
➜  Repos/work  sudo !!                                                                                           
➜  Repos/work  sudo mkdir -p /etc/docker/certs.d/docker-registry.foodomain.org                                    
➜  Repos/work  cp -v ~/Downloads/domain-cacert.pem /etc/docker/certs.d/docker-registry.foodomain.org/ca.crt     
cp: failed to access '/etc/docker/certs.d/docker-registry.foodomain.org/ca.crt': Permission denied                
➜  Repos/work  sudo !!                                                                                           
➜  Repos/work  sudo cp -v ~/Downloads/domain-cacert.pem /etc/docker/certs.d/docker-registry.foodomain.org/ca.crt
'/home/dminca/Downloads/domain-cacert.pem' -> '/etc/docker/certs.d/docker-registry.foodomain.org/ca.crt'        
➜  Repos/work  sudo service docker restart                                                                       
➜  Repos/work  docker login docker-registry.foodomain.org                                                         
Username: dminca@foodomain.com                                                                        
Password:                                                                                                        
Login Succeeded