Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Allow insecure ssl #2687

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet

dvoet commented Nov 13, 2013

This will allow support for private docker registries that have self-signed or otherwise unverified ssl certificates. Useful in non-production environments. As per prior pull request comments, this is implemented using commandline arguments to the push, pull, import and login commands. It was not implemented in the run command as it did not fit in nicely with the parameter parsing. In order to run images from a private registry it must be explicitly pulled first.

Collaborator

vieux commented Dec 2, 2013

ping @shin-

dvoet commented Dec 14, 2013

anyone have any comments on this?

Contributor

SvenDowideit commented Dec 14, 2013

can this code and the client portion from #3068 be combined?

dvoet commented Dec 17, 2013

personally, I think this stands on it own. All private repos using basic auth need to run with ssl (at least that is how it is currently in written). This pull request makes it so that one only needs purchase certs for production environments.

Contributor

creack commented Jan 3, 2014

@dvoet Can you rebase?

Contributor

creack commented Jan 3, 2014

@shin- can you take a look?

Contributor

shin- commented Jan 9, 2014

Needs to be rebased, otherwise LGTM

@shin- shin- referenced this pull request in docker/docker-registry Jan 9, 2014

Closed

Docs mention basic auth, but client does not know how #82

bumping this thread to hopefully get this rebased. I'm writing up a blog post on using the registry in production. It would be awesome to have this PR in so that it can be demonstrated with both self-signed certs and CA certs. :)

dvoet commented Jan 14, 2014

working on it now

On Mon, Jan 13, 2014 at 8:07 PM, Matthew Fisher notifications@github.comwrote:

bumping this thread to hopefully get this merged. I'm writing up a blog
post on using the registry in production. It would be awesome to have this
PR in so that it can be demonstrated with both self-signed certs and CA
certs. :)


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/pull/2687#issuecomment-32229866
.

@dvoet @dvoet dvoet + dvoet implement switch to allow insecure ssl
Docker-DCO-1.1-Signed-off-by: Douglas Voet <doug.voet@gmail.com> (github: dvoet)
aeea586

dvoet commented Jan 14, 2014

all set :)

dvoet commented Jan 28, 2014

ping

webts commented Jan 30, 2014

+1

+1

+1

Rebase pls?

dvoet commented Feb 5, 2014

Come on guys. You asked me to rebase a month ago and I did so. I don't have time to do it again right now.

Contributor

jamtur01 commented Feb 5, 2014

Can we get another maintainer +1 on this - @creack @vieux @crosbymichael ? Thanks!

Could we please get an update to when this is likely to be included... was really hoping it was in 0.8

djsell commented Feb 18, 2014

I would love to see this make it in. Thanks!

Member

thaJeztah commented Feb 27, 2014

+1 sorry for +1-ing, but it seems this issue may be forgotten otherwise?

Contributor

ewindisch commented Feb 27, 2014

This is the equivalent of adding an "ignore and continue" button to a browser's SSL warning. Browsers got rid of those years ago and for good reason.

I'd prefer we just told people how to update their OS' certificate files.

    "/etc/ssl/certs/ca-certificates.crt", // Linux etc
    "/etc/pki/tls/certs/ca-bundle.crt",   // Fedora/RHEL
    "/etc/ssl/ca-bundle.pem",             // OpenSUSE
    "/etc/ssl/cert.pem",                  // OpenBSD
    (KeyChain on MacOS)
    (Windows - http://support.microsoft.com/kb/295663)

dvoet commented Feb 27, 2014

I don't think browsers got rid of this. Chrome certainly has a big scary warning but it has an ignore button. Curl certainly has the same functionality with the -k switch. And not everyone has access to change their cert file.

Member

thaJeztah commented Feb 27, 2014

@ewindisch agreed; documenting proper ways to achieve the same goal should be preferred. Is there already an issue to add that to the documentation? This would enable the maintainers to close this issue as. 'Won't fix, doing issue xxx in stead'

Contributor

ewindisch commented Feb 27, 2014

@thaJeztah I've created an issue #4372 for documenting local CA management.

@dvoet - It's obviously non-consistent between browsers and some have gotten more restrictive then loosened up again. Arguably if you don't have permission to change your cert file, you shouldn't be ignoring SSL certificates. I'm not really fond of the fact that other clients have this setting, it's too frequently abused. However, putting an interactive prompt on this that prevents automation would be a good compromise from my perspective.

Member

thaJeztah commented Feb 27, 2014

@ewindisch thanks!

Contributor

crosbymichael commented Mar 27, 2014

I agree with @ewindisch on this one. I think this should be a doc change and not a code change.

@shykes @creack @unclejack what do you think?

@ewindisch Even putting aside the no-so-great trust model based on CAs, the consequence of not being able to easily ignore a self-signed certificate (used a lot in dev) is that many people will just use http instead. After all http does not check identities and it's also open to man in the middle attacks. https with a signed certificate at least solves the later problem.
Thus, I would guess leaving out a flag like curl's -k will in effect lower the security of private Docker repos.

I think it should be a code change.

Collaborator

shykes commented Apr 18, 2014

Since this doesn't seem unanimous among the security-conscious, in doubt I'm going to go for the conservative choice. So closing this.

My apologies @dvoet, we should not have given you all this work before being sure the design itself was agreed upon. We'll be more careful in the future.

@shykes shykes closed this Apr 18, 2014

kennu commented May 7, 2014

I just want to note that it's very difficult to debug why Docker won't connect to our private PositiveSSL-certificate image registry (which works fine with curl, Safari, Chrome, etc). The registry has thus been unusable for weeks, because there is no way to force the Docker client to ignore the SSL verification. I would really have preferred to have an override switch to let our workflow continue.

Contributor

ewindisch commented May 7, 2014

@kennu - why can you not add the PositiveSSL CA Bundle to your host? What makes it hard to debug? There might be a bug that failures are not clearly presented (related to improving the documentation on how to fix/address these issues).

(download the bundle here: https://support.comodo.com/index.php?_m=downloads&_a=view&parentcategoryid=4)

kennu commented May 7, 2014

It's hard to debug, because everything else (curl, browsers) works fine except the Docker client, and the only error from Docker is "HTTP code 401, Docker will not send auth headers over HTTP." So basically we couldn't spend days debugging this and started using "docker save | ssh .. docker load" instead.

I can still try to debug the actual problem. My point is that a certificate issue like this will halt the whole private registry workflow, because there is no "emergency override" when something breaks.

Contributor

ewindisch commented May 7, 2014

@kennu - The primary issue for you seems to be that the error is obscure and unhelpful.

Contributor

shin- commented May 7, 2014

I can provide some insight on this

"Docker will not send auth headers over HTTP."

This specific message has been added semi-recently by #5083. When the docker client is told to connect to a private registry, it will always try to connect over HTTPS first, and default to HTTP afterwards if the /v1/_ping response hasn't been satisfactory (see ExpandAndVerifyRegistryUrl in registry/registry.go). In your case, this fallback happens, but your registry is asking for credentials over HTTP, which docker deliberately won't provide, because basic auth without SSL is a bad idea.

You should investigate what response you're getting over HTTPS from your registry when GETting /v1/_ping, and go from there. Let us know if we can help further.

kennu commented May 7, 2014

@ewindisch That, and also the fact that Docker runs half in boot2docker and half in OS X, so it's hard to say where exactly the real problem is. It might also be an issue that will be fixed in a later Docker or boot2docker release. During which time I would have liked to use an SSL verification override switch to continue working.

kennu commented May 7, 2014

@shin- I can see Docker sending this for the first https connection attempt (in Wireshark):

TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Bad Certificate)

So I assume that the certificate is just not working. Debugging is complified by the fact that another guy changed the Apache2 certificate and CA chain bundle (after Heartbleed), so I don't know all details, and now it just works everywhere except in Docker..

kennu commented May 7, 2014

By the way, Docker gives a different error message when using "docker login" (than "docker push"):

Error: Invalid Registry endpoint: Get https://example.com/v1/_ping: x509: certificate
signed by unknown authority (possibly because of "x509: cannot verify signature:
algorithm unimplemented" while trying to verify candidate authority certificate
"COMODO RSA Certification Authority")

Any ideas about the unimplemented algorithm? OpenSSL shows this output with s_client (I changed our domain to example.com):

Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL Wildcard/CN=*.example.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA     Domain Validation Secure Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

Continued:

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: xxx
    Session-ID-ctx: 
    Master-Key: xxx
    Key-Arg   : None
    Start Time: 1399484769
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

kennu commented May 7, 2014

It sounds like my error is probably caused by #5600. Some of the PositiveSSL intermediate certificates are using Signature Algorithm: sha384WithRSAEncryption, which apparently requires the import sha512 clause.

So after all, I'm not sure if this kind of an error could even be overridden with a "skip SSL verification" option.

echa commented May 7, 2014

I had the same issue in another project. The solution is to include all crypto hash families, i.e.

import (
    _ "crypto/md5"    // make MD5 hash available for SSL signature check
    _ "crypto/sha256" // make SHA224/256 hashes available for SSL signature check
    _ "crypto/sha512" // make SHA384/512 hashes available for SSL signature check
)

Otherwise the algorithms don't get linked into the static binary.

kennu commented May 7, 2014

My problem was just fixed by building the current docker-0.11.0-dev from source and using it inside boot2docker. Now the certificate is accepted. Thanks to everybody for the support!

Member

thaJeztah commented May 7, 2014

@kennu from the release notes:

Docker now works with registries that support SHA-512

Sounds like that is what solved it for you

jbladen commented Jun 12, 2014

Can we widen that to include those root certs which use MD5?

Any updates on this ? Still getting the "Error response from daemon : Invalid Registry endpoint : Get https://localhost/v1_ping: x509: Certificate signed by unknown authority.". Or any clear workarounds for this ?

jbladen commented Jul 23, 2014

In what version? This issue was resolved by the patch for my sha-384 cert.
On 23 Jul 2014 02:33, "Shankar" notifications@github.com wrote:

Any updates on this ? Still getting the "Error response from daemon :
Invalid Registry endpoint : Get https://localhost/v1_ping: x509:
Certificate signed by unknown authority.". Or any clear workarounds for
this ?


Reply to this email directly or view it on GitHub
dotcloud#2687 (comment).

Client version: 1.1.0
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): 79812e3
Server version: 1.1.0
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): 79812000

I generated the certificate following this page : http://www.activestate.com/blog/2014/01/deploying-your-own-private-docker-registry

nikvdp commented Jul 27, 2014

For other googlers who end up here, here's how I got my docker registry working with a self signed certificate (mostly cobbled together from here and here). The key is to act as your own CA.

Not suitable for production, but at least it allows you to test out a dev docker registry. On a private server do the following:

Generate a new root key:

openssl genrsa -out devdockerCA.key 2048

Generate a matching root cert:

openssl req -x509 -new -nodes -key devdockerCA.key -days 10000 -out devdockerCA.crt

Generate your server key:

openssl genrsa -out dev-docker-registry.com.key 2048

and cert request (make sure to type in the external name/ip (w/out port num) of your server when prompted for 'Common Name':

openssl req -new -key dev-docker-registry.com.key -out dev-docker-registry.com.csr

Then sign it:

openssl x509 -req -in dev-docker-registry.com.csr -CA devdockerCA.crt -CAkey devdockerCA.key -CAcreateserial -out dev-docker-registry.com.crt -days 10000

Take the dev-docker-registry.com.crt and dev-docker-registry.com.key files and copy them to your docker-registry server. Then configure nginx to point to this ssl key and cert as normal. (probably something like this):

 server {
      listen 443;
      server_name my-dev-docker-registry.com;

      ssl on;
      ssl_certificate /etc/ssl/certs/dev-docker-registry.com.crt;
      ssl_certificate_key /etc/ssl/dev-docker-registry.com.key;
      ...
}

Take the devdockerCA.crt from the first step and copy it to each client machine you plan to use with this docker-registry instance. Copy it to the machine's certs folder and update certs as follows (this step is debian/ubuntu specific, for other platforms see ewindisch's comment above):

sudo mkdir /usr/local/share/ca-certificates/docker-dev-cert
cp devdockerCA.crt /usr/local/share/ca-certificates/docker-dev-cert
sudo update-ca-certificates

after restarting nginx you should be able to push and pull to your docker registry over self-signed SSL now.

@ewindisch if there's a more appropriate place to post this let me know, would be glad to clean this up and properly add it to the documentation.

zedtux commented Aug 22, 2014

@nikvdp I have followed your instructions (excepted that I'm on OS X) and it's not working, I still have the 'x509: certificate signed by unknown authority' error.

The only thing I did different in order to import my certificate on my Mac is that I have opened it, requested OS X to trust the certificate and entered my password.
Using Safari it is working perfectly (and the certificate is shown as trusted) but with the docker login command it's failing.

$ curl -u zedtux:xxxxxxx https://docker.zedroot.org
"docker-registry server (prod)"%
$ curl -u zedtux:xxxxxxx https://docker.zedroot.org/v1/_ping
true%

@zedtux, @shin- I decided not to use Mac OS X since I do not use it in production. I setup an ubuntu vm using virtualbox and if you want some starting point you should go to:

https://help.ubuntu.com/12.04/serverguide/certificates-and-security.html

Then, once you create your own CA and sign your key, make sure sure to add the ca cert on your client machine. If you want login/logout functionality, use docker 1.2.0 or use coreos alpha release.

curl - u user:pass and docker login/logout works and credentials are saved at home dir,

But, I can't push images and get HTTP 401 error and looking at nginx 1.4.X logs, says no credentials provided. It's using ssl to connect but docker client is not sending them. I'll use wireshark or tcpdump to look at the traffic and provide results.

Error: docker/docker-registry#298

solved... just do not forget to start docker using ca cert ;) (facepalm) I know

https://docs.docker.com/articles/https/

Contributor

shin- commented Sep 8, 2014

Glad you were able to solve it, thanks for reporting back!

zedtux commented Sep 8, 2014

Thank you @jonathanve ! I will try this.

matleh commented Sep 11, 2014

The docker daemon itself supports the use of self-signed certificates with "--tlsverify --tlscacert=ca.pem". I am having a hard time understanding, why it would be such a serious security flaw to support the same thing (like a '--tlscacert` param for docker pull/push/login) forthe communication between docker and docker-registry.

I am trying way too long already to get this working. Seems a bit like Docker wants to force users into using the Docker Hub by making the operation of a private registry (which is secure and supports some kind of authentication) a very intricate endeavor.

@matleh matleh referenced this pull request in docker/docker-registry Sep 15, 2014

Open

Authentication on private registry #541

@jonathanve I have encountered the same problem with you. First HTTP 401 error, after added ssl certificate, then unknown authority. Restarting docker daemon with cacert.pem set me free!

@seanlook awesome you are now using your private docker index!

Not sure if this should be posted here but this is not clear to me after going through a lot of topics.
Does using a self signed certificate is working or not to connect to a private registry behind nginx ?

I have followed steps mentioned by @nikvdp with no luck with both version 1.3.0 and 1.3.2 based on #8467

Client version: 1.3.0
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): c78088f 
OS/Arch (client): linux/amd64
Server version: 1.3.0
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): c78088f

~$ sudo docker login -u user -p pwd https://10.4.5.5:443

[debug] endpoint.go:54 Registry https://10.4.5.5:443/v1/ does not work (Get    https://10.4.5.5:443/v1/_ping: x509: cannot validate certificate for 10.4.5.5 because it doesn't contain any IP SANs), falling back to http

Seems like I got same issue with 1.3.2 binaries even when using --insecure-registry parameter but at least I got the message on the client side.

Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa

~$ sudo ./docker -d -D --insecure-registry 10.4.5.5:443
~$ sudo docker login -u user -p pwd https://10.4.5.5:443

Error response from daemon: Server Error: Post https://10.4.5.5:443/v1/users/: x509: cannot validate certificate for 10.4.5.5 because it doesn't contain any IP SANs

I also tried the same with no --insecure-registry and ca.crt at the required place with quite same result.

Error response from daemon: Invalid registry endpoint https://10.4.5.5:443/v1/: Get https://10.4.5.5:443/v1/_ping: x509: cannot validate certificate for 10.4.5.5 because it doesn't contain any IP SANs. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add ``--insecure-registry 10.4.5.5:443`` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/10.4.5.5:443/ca.crt

Hi @jncharpin , hope you are ok. This is the process I followed to use a self signed certificate and a private registry behind nginx:

  1. Create an amazon ec2 instance and assign a domain name to its IP address.
  2. I self signed the certificate using the domain name, not IP address following:
    https://help.ubuntu.com/12.04/serverguide/certificates-and-security.html
  3. Setup private registry and nginx (using ssl)
  4. Add the ca cert on your client(s) machines(s)
  5. Add entry to /etc/hosts (ip hostname)
  6. Restart docker service
  7. Login to private registry
  8. Push Images to It.

And it works.

nikvdp commented Nov 25, 2014

@jncharpin I wrote a more detailed writeup of how to get this working over at digitaloceans community site, might help you out.

Short answer: using a self signed cert works fine, and can even work without the --insecure-registry flag if you set up your CA certs properly. As @jonathanve pointed out, restarting the docker service on the client after adding the cert is essential as otherwise docker won't pick up the newly authorized certs.

Collaborator

tiborvass commented Nov 25, 2014

@nikvdp Maybe I'm wrong, but in my tests, I didn't need to restart the service. Seems to me Docker reads the cert file every single time: https://github.com/docker/docker/blob/662efd95c7a30e5593ca9fdc672ad3f1d6f9703c/registry/registry.go#L107 in doRequest.

nikvdp commented Nov 25, 2014

@tiborvass could be i'm mistaken on that bit. I went through this for a few registries tho, and was never able to get it to work until after I had restarted the docker service on the client. To be clear, I was adding a CA cert that had been used to sign my self-signed certificate to the client's store via ubuntu's update-ca-certificates cmd, and not using the --insecure-registry flag, the goal being to fool docker into thinking my self-signed cert was signed by a valid authority and sidestep the issue.

not sure what's in the /etc/docker/certs.d mentioned there (and don't see it on my system), but as far as I could tell docker wasn't re-checking the list of valid CA certs until with each request (which makes sense as those should very rarely change).

Thanks all for your reply.

@nikvdp this is from your article on digital ocean that I setup the registry and ssl. I restarted the service many times and even tried different version of the service.
I followed exact same step to self signed the certificate and update it. Both service and registry are on same ubuntu machine.

From my understanding docker is considering the certificate but it seems to fail in x509 which I'm not familiar with.

Regarding @jonathanve comment I'm wondering if using an IP as domain name could make a difference.

Collaborator

tiborvass commented Nov 25, 2014

@nikvdp oh right that makes sense. If you put the cert under /etc/docker/certs.d/registryname:port/ca.crt it will get added to the pool, in addition to the system root certs. I think changing the root certs need a docker restart, but not if the cert is under /etc/docker/certs.d.

nikvdp commented Nov 26, 2014

@tiborvass good to know! will try that out next time.

@jncharpin I think you're right about the problem having to do with IP vs domain name. I've never tried it using an IP directly. What are you entering in the common name field portion when you do the certificate signing steps?

I'd try making a new certificate and using a domain name instead of the ip for your registry, making sure that you enter the domain name of your registry into openssl's Common Name prompt instead of the ip when you sign the cert. If you don't have a domain then you can just fake it as @jonathanve pointed out by adding an entry into your /etc/hosts file on the client machine. As long as the "domain" of your registry and the common name of the signed cert match everything should work.

@nikvdp that is correct. In fact, the message "[...] it doesn't contain any IP SANs" appears if you use the ip address. Trust me

@jonathanve @nikvdp I can also confirm that the issue is coming from the usage of the IP instead of a domain name in the certificate.
This is working fine now thanks a lot for your support and the great tutorial.

@jncharpin No problem. Would be nice to write a chef cookbook for it. If you want my help, just let me know,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment