Docker registry v2 TLS with Letsencrypt gave x509: certificate signed by unknown authority

[charliemaiors]

Description

I've tried to start a docker registry as container running on a private machine with dynamic dns, I've configured certbot on the host machine in order to obtain certificates from letsencrypt and started the registry using this command

docker run -d -p 443:5000 --restart=always --name registry -v /etc/letsencrypt/archive/<my-domain>:/certs -v /opt/registry-auth/:/opt/registry-auth -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/cert1.pem -e REGISTRY_HTTP_TLS_KEY=/certs/privkey1.pem -e HTTP_TLS_LETSENCRYPT_CACHEFILE=/root/letsencrypt -e HTTP_TLS_LETSENCRYPT_EMAIL=<my-reg-mail> -e REGISTRY_HTTP_SECRET=nicesecret -e REGISTRY_AUTH_HTPASSWD_REALM=basic-realm -e REGISTRY_AUTH_HTPASSWD_PATH=/opt/registry-auth/dockerauth registry:2

Notes

• In /opt/registry-auth/ is the htpasswd file
• With /etc/letsencrypt//live folder I've just links so no valid certs

when I try to interact with registry (login, push) the response is always x509: certificate signed by unknown authority

Steps to reproduce the issue:

1. Configure new host and download latest docker engine
2. Configure a dynamic dns for that host, obtain with certbot ssl certificates
3. Run docker registry with command specifed above and try to login or push

Describe the results you received:

Docker login
docker login -u <user> -p <password> <my-domain>

Result
Error response from daemon: Get https://<my-domain>/v1/users/: x509: certificate signed by unknown authority

Docker push
docker tag java <my-domain>/java docker push <my-domain>/java

Result
Get https://<my-domain>/v1/_ping: dial tcp 137.204.57.31:443:x509: certificate signed by unknown authority

Output of docker version:

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:07:40 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:07:40 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:07:40 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:07:40 2017
 OS/Arch:      linux/amd64
 Experimental: false
pmaiorano@lab2-OptiPlex-780:~$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 5
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.8.0-39-generic
Operating System: Ubuntu 16.10
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.668 GiB
Name: lab2-OptiPlex-780
ID: QNEG:L2CC:IJR5:ZQSR:SHHN:WB2F:LETL:VFTU:Z3AG:QLXL:SG2H:O4N6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical host on Ubuntu 16.10 Desktop environment

[thaJeztah]

/cc @cyli @dmcgowan

[dmcgowan]

If you are using the let's encrypt feature built into the registry the environment variable is REGISTRY_HTTP_TLS_LETSENCRYPT_CACHEFILE and no certificate should be provided. If you are still having issues with the Let's Encrypt support on the registry please open up an issue here https://github.com/docker/distribution

[bkcsfi]

If someone else stumbles across this, I suspect the issue maybe related to the certificate being used:

REGISTRY_HTTP_TLS_CERTIFICATE=/certs/cert1.pem

I think you probably want to use fullchain.pem instead of cert.pem because neither docker (go lib) nor (ubuntu in my case) have LE root cert built in at this time

See also #8849 (comment)

I have gitlab setup with LE certificate. Browser works fine, but docker fails to push to registry.

Changing to fullchain.pem as the certificate (.crt) resolved the issue

[scottstensland]

I feel above comment is misleading ... cert.pem from letsencrypt should work fine as its the public key and works fine for me using a local docker registry ... avoid throwing around fullchain.pem as that contains your private key which may work as a replacement for cert.pem however doing so may let the cat out of the bag ;-(

UPDATE ... I stand corrected

ssl_certificate which will point to the certificate chain   fullchain.pem    
ssl_certificate_key which will point to the private key     privkey.pem

https://serverfault.com/questions/793474/how-to-get-lets-encrypt-public-private-key

[ijc]

avoid throwing around fullchain.pem as that contains your private key

I don't think it should, it should contain your cert + any intermediate certs between that and the CA and (perhaps or always?) the actual CA cert too i.e. a bunch of certs only. It wouldn't normally include any keys, they would normally be in a separate foo.key file.

If something is embedding keys in a file named fullchain.anything I think a bug report against that tool would be in order since that is both dangerous (as you say) and unexpected.

[alexivkin]

Fullchain is the right solution to this problem. The reason is slightly different. Let's Encrypt root CA is in most cert stores on most OSes by now, but the intermediate CA may not be, because LE updates intermediates sometimes. Your cert is signed with the intermediate, and if you don't provide the full chain - from the cert to the root, SSL lib does not know which root to verify with and fails.

avoid throwing around fullchain.pem as that contains your private key

Full chain does not contain the private key. In fact you can't even add if you wanted, because it will no longer be a valid x509.

[denikaryadi-2606]

Hi Bro.. This issue same as with my problem.
Openshift cannot import-image for nexus repository, the sintax is
oc import-image nexus-coba:3.5 --from=192.168.250.250:8083/node-nexus --confirm
error: tag latest failed: Internal error occurred: Get https://192.168.250.250:8083/v2/: x509: certificate signed by unknown authority
imagestream.image.openshift.io/nexus-coba imported with errors
This Solution only add --insecure after --confirm.

oc import-image nexus-coba:3.5 --from=192.168.250.250:8083/node-nexus --confirm --insecure

[denikaryadi-2606]

Hi Bro.. This issue same as with my problem.
Openshift cannot import-image for nexus repository, the sintax is
oc import-image nexus-coba:3.5 --from=192.168.250.250:8083/node-nexus --confirm
error: tag latest failed: Internal error occurred: Get https://192.168.250.250:8083/v2/: x509: certificate signed by unknown authority
imagestream.image.openshift.io/nexus-coba imported with errors
This Solution only add --insecure after --confirm.

oc import-image nexus-coba:3.5 --from=192.168.250.250:8083/node-nexus --confirm --insecure