SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

[gkostyanikov]

Got this error on both machines almost at the same time with docker-compose and lately with fig after rollback. A few search results points to python/openssl issue but i simple can't figure out where to dig to. Python/openssl comes from homebrew.

Boot2Docker-cli version: v1.4.1
Git commit: 43241cb

Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.4
Git commit (client): 5bc2ff8
OS/Arch (client): darwin/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

[adambiggs]

I think I'm experiencing the same thing trying to use the docker-compose release candidate...

$ docker-compose ps
SSL error: hostname '192.168.59.103' doesn't match 'boot2docker'

But fig works fine...

$ fig -f docker-compose.yml ps
Name   Command   State   Ports
------------------------------

I'm on OSX, running all the same versions as @gkostyanikov, except my Go client version is go1.3.3. My python/openssl is also installed via Homebrew. Might have something to do with it?

Edit: Actually it looks like Homebrew doesn't link openssl, so I'm using the default OSX version: OpenSSL 0.9.8za 5 Jun 2014.

[adambiggs]

The issue was Homebrew python.

docker-compose now works after uninstalling homebrew python/openssl, installing pip with easy_install, and reinstalling docker-composer using the system python.

[gkostyanikov]

@adambiggs Your solution works! Thanks!

[keekdageek]

This worked for me too, I'm using a brand new mac and set it up with homebrew python. Had this error with fig communicating with docker. Followed @adambiggs advice verbatim and got past my blocker, it could be a python version issue too but regardless I guess this machine will be using system python for awhile.

[zkanda]

This is happening on me too. And I don't want to use the system's python, anyone have another workaround?

[aanand]

Have you tried using the binary? Do you get the same problem?

[zkanda]

No I haven't tried the binary.
If you don't want to install it in your systems python, another workaround is to use virtualenv(wrapper).

mkvirtualenv --python=/usr/bin/python docker-compose
pip install docker-compose==1.1.0-rc2

[adambiggs]

Found a better solution using pyenv to roll back to python 2.7.8:

http://stackoverflow.com/a/28216459/1166293
https://github.com/yyuu/pyenv

Edit: Nevermind, pyenv introduced a bunch of it's own problems...

[ocasta]

What caused this error for me was that the home-brew openssl was not linked to /usr/local/bin/openssl.

openssl version

returned OpenSSL 0.9.8zc 15 Oct 2014 not OpenSSL 1.0.1j 15 Oct 2014

Running

brew link --force openssl

and reinstalling fig resolved the issue.

[zkanda]

Interesting, however my OpenSSL version is OpenSSL 1.0.1j 15 Oct 2014

[andizzle]

@aanand in my case the binary does not have this problem.

[NotBobTheBuilder]

I got this error when I had fig installed through pip, not homebrew. sudo pip uninstall fig and brew install fig fixed it for me.

[szeryf]

+1 for @NotBobTheBuilder solution, also worked for me

[octplane]

👍 for @NotBobTheBuilder

[adambiggs]

@NotBobTheBuilder nice solution for fig but docker-compose isn't available on homebrew yet unfortunately.

[adambiggs]

@ocasta what about this scary-sounding warning from homebrew about linking OpenSSL?

This formula is keg-only.
Mac OS X already provides this software and installing another version in
parallel can cause all kinds of trouble.

Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries

[mewm]

Thumbs up @NotBobTheBuilder - That fixed it with me as well.

[anentropic]

anyone know the source of this problem? it's happening to me with fig. I prefer to stick to pip install fig like I have now. It all worked fine a couple of weeks ago, don't know what's changed on my system

[anentropic]

My system OpenSSL is OpenSSL 0.9.8zc 15 Oct 2014, my homebrew openssl is newer but not linked.

...I am guessing it broke when I upgraded to Python 2.7.9, there seem some SSL related bugs with it... looks a lot like this:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196431
http://bugs.python.org/issue23052

[anentropic]

running brew link --force openssl and reinstalling fig did not do anything for me.

[anentropic]

Does fig need updating to work around the SSL changes in Py 2.7.9?
https://www.python.org/dev/peps/pep-0476/#opting-out

[anentropic]

I'm using boot2docker. I just upgraded to 1.5.0 but no change.

In [1]: from fig.cli.docker_client import docker_client

In [2]: client = docker_client()

In [3]: client.version()

SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

In [4]: %debug
> /Users/anentropic/.virtualenvs/dpm/lib/python2.7/site-packages/requests/sessions.py(461)request()
    460         send_kwargs.update(settings)
--> 461         resp = self.send(prep, **send_kwargs)
    462

ipdb> p settings
{'verify': '/Users/anentropic/.boot2docker/certs/boot2docker-vm/ca.pem', 'cert': ('/Users/anentropic/.boot2docker/certs/boot2docker-vm/cert.pem', '/Users/anentropic/.boot2docker/certs/boot2docker-vm/key.pem'), 'proxies': {}, 'stream': False}

The fig code looks correct, it is attempting to use the certs installed by boot2docker... I assume these certs are ok because they always used to work and I just upgraded b2d so they shouldn't have expired.

[anentropic]

Hmm, my Python (installed via homebrew) appears to be using the homebrew version of OpenSSL though:

$ python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.0.2 22 Jan 2015

$ brew info openssl
openssl: stable 1.0.2 (bottled)
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /usr/local/etc/openssl/certs

and run
  /usr/local/opt/openssl/bin/c_rehash

...running /usr/local/opt/openssl/bin/c_rehash didn't help :)

[kretz]

I tried a previously installed version of python (2.7.8_2) via $ brew switch python 2.7.8_2 with the same problem (even if the error message was slightly different). So the python 2.7.9 version seems not to be the problem.

Then I tried switching to an older openssl version, from 1.0.2 to 1.0.1j_1 which seems to work.

$ python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.0.2 22 Jan 2015
$ docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
$ brew switch openssl 1.0.1j_1
$ python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 1.0.1j 15 Oct 2014
$ docker-compose ps
Name   Command   State   Ports 
------------------------------

[anentropic]

For me I just get a different error, but maybe it helps narrow down what's wrong:

$ brew switch openssl 1.0.1j_1
Error: openssl does not have a version "1.0.1j_1" in the Cellar.
Versions available: 1.0.1e, 1.0.1f, 1.0.1g, 1.0.2
$ brew switch openssl 1.0.1g
Opt link created for /usr/local/Cellar/openssl/1.0.1g
$ fig up
SSL error: hostname '192.168.59.103' doesn't match 'boot2docker'

Switching back to OpenSSL 1.0.2 produces the previous CERTIFICATE_VERIFY_FAILED error so changing versions definitely has some effect

[cydparser]

One workaround is to run docker-compose in a container:

git clone git@github.com:docker/fig.git
cd fig
docker build --tag docker-compose .

alias docker-compose='docker run --rm -e "DOCKER_TLS_VERIFY=$DOCKER_TLS_VERIFY" -e DOCKER_HOST=tcp://172.17.42.1:2376 -e DOCKER_CERT_PATH=/usr/local/certs -v "$DOCKER_CERT_PATH:/usr/local/certs" -v "$PWD:/code" docker-compose --project-name "${PWD##*/}"'

This requires exposing port 2376 in VirtualBox:

VBoxManage controlvm boot2docker-vm natpf1 "docker-s,tcp,127.0.0.1,2376,,2376"

[datariot]

@kretz's answer worked for me.

[boostrack]

+1 @kretz brew switch openssl 1.0.1j_1
made the trick

[mattattui]

@jmmills huh… same here. Maybe python treats set-as-empty differently to unset?

Mac OS, homebrew docker-compose and docker-machine, using system python. Newly created machine with: docker-machine create --driver=vmwarefusion --vmwarefusion-memory-size 1536 dev

env | grep CURL returns nothing
docker-compose ps returns

ERROR: SSL error: hostname '172.16.129.133' doesn't match 'localhost'

CURL_CA_BUNDLE='' docker-compose ps returns:

/usr/local/Cellar/docker-compose/1.7.0/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/usr/local/Cellar/docker-compose/1.7.0/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Name   Command   State   Ports 
------------------------------

[alexwilson]

I've had the exact same - CURL_CA_BUNDLE was not being set in my env, and setting it to an empty string gave me the same output as @inanimatt.

[jmmills]

It definitely smells like an upstream bug, my guess would be some environment compatibility code for curl, in which "defined" and "empty" are being treated differently.

Thanks,
Jason Mills

• sent from mobile.

On Apr 24, 2016, at 6:14 AM, Alex Wilson notifications@github.com wrote:

I've had the exact same - CURL_CA_BUNDLE was not being set in my env, and setting it to an empty string gave me the same output as @inanimatt.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[mattattui]

Only seems to affect the homebrew version - installing homebrew Python then installing docker-compose via pip solves all the errors.

On 24 Apr 2016, at 14:14, Alex Wilson notifications@github.com wrote:

I've had the exact same - CURL_CA_BUNDLE was not being set in my env, and setting it to an empty string gave me the same output as @inanimatt.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[jmmills]

I believe I pasted replication of the issue in Linux earlier. I can double check tomorrow when at a workstation

Thanks,
Jason Mills

• sent from mobile.

On Apr 24, 2016, at 12:22 PM, Matt Robinson notifications@github.com wrote:

Only seems to affect the homebrew version - installing homebrew Python then installing docker-compose via pip solves all the errors.

On 24 Apr 2016, at 14:14, Alex Wilson notifications@github.com wrote:

I've had the exact same - CURL_CA_BUNDLE was not being set in my env, and setting it to an empty string gave me the same output as @inanimatt.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[nanoz]

Same problem here since i updated docker-compose to version 1.7 using brew.

$ docker-compose ps
ERROR: SSL error: hostname '192.168.99.100' doesn't match 'localhost'

$ docker-compose version
docker-compose version 1.7.0, build unknown
docker-py version: 1.8.0
CPython version: 2.7.10
OpenSSL version: OpenSSL 0.9.8zh 14 Jan 2016

Emptying the CURL_CA_BUNDLE env var kind of solves the problem :

CURL_CA_BUNDLE= docker-compose ps
/opt/boxen/homebrew/Cellar/docker-compose/1.7.0/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/opt/boxen/homebrew/Cellar/docker-compose/1.7.0/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
/opt/boxen/homebrew/Cellar/docker-compose/1.7.0/libexec/vendor/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
   Name                 Command               State    Ports
------------------------------------------------------------

Downgrading to 1.6.2 also solves the problem.

$ brew switch docker-compose 1.6.2
Cleaning /opt/boxen/homebrew/Cellar/docker-compose/1.4.2
Cleaning /opt/boxen/homebrew/Cellar/docker-compose/1.5.1
Cleaning /opt/boxen/homebrew/Cellar/docker-compose/1.5.2
Cleaning /opt/boxen/homebrew/Cellar/docker-compose/1.6.0
Cleaning /opt/boxen/homebrew/Cellar/docker-compose/1.6.2
Cleaning /opt/boxen/homebrew/Cellar/docker-compose/1.7.0
3 links created for /opt/boxen/homebrew/Cellar/docker-compose/1.6.2

$ docker-compose ps
   Name                 Command               State    Ports
------------------------------------------------------------

[aboutlo]

Rather than disable the CURL_CA_BUNDLE you can run using:
CURL_CA_BUNDLE=~/.docker/machine/machines/default/ca.pem docker-compose ps

[jmmills]

I'm probably not the first one who may have brought this up, but isn't it counter intuitive that a curl environment variable have any effect what so ever on an unrelated Python application?

Thanks,
Jason Mills

• sent from mobile.

On May 7, 2016, at 3:22 PM, Lorenzo Sicilia notifications@github.com wrote:

Rather than disable the CURL_CA_BUNDLE you can run using:
CURL_CA_BUNDLE=~/.docker/machine/machines/default/ca.pem docker-compose ps

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[m-housh]

I ran into this issue and the problem was with the environment variable REQUESTS_CA_BUNDLE pointing to a custom location for self-signed certs. Encase this helps anyone.

• Michael Housh

[iongion]

@aboutlo This works - it did not work with other ca.pem file, only with this one. I am also on Windows so I have a more voodoo configuration, thank you!

[boxsterman]

Uninstalling ndg-httpsclient (with pip - was version 0.4.0) solved the issue for me, see my post here: #3365

[jitekuma]

I debugged docker-compose and docker-py and figured out that you should either use environment variables or options in command. You should not mix these . If you even specify --tls in command then you will have to specify all the options as the TLSConfig object, as now TLSConfig object is created completely from the command options and operide the TFSConfig object created from the environment variable.

[AndreLouisCaron]

@m-housh OMG thanks for that tip! Exact same thing happened to me! Removed REQUESTS_CA_BUNDLE from my environment and it resolved this issue for me.

[mesutayata]

I have encountered the same problem. First I though it is because the OpenSSL version differences (Pyhton had 1.0.2 but OS had 0.9.8) I made them both 1.0.2 but it still did not work.
I have solved my problem by simply ssh to docker and then check my certificate in authorized keys and update it. Interestingly somehow it was a wrong certificate there.

Follow these steps please:

boot2docker ssh
docker@boot2docker:~$ cat .ssh/authorized_keys

Check if this certificate is really the certificate from your computer. If not just copy yours in to this file and save it. Then just run:

docker-compose up

This worked for me and hope that it helps.

[ijc]

Issue grooming: There appears to be a variety of different failure modes and user error/misconfiguration scenarios (all largely historic) described here.

I'm not seeing anything which seems to point towards an active ongoing issue in compose, so I'm closing the issue. If you are still seeing a related error with modern versions then please open a fresh issue with full details of your scenario etc.