problem with Python 2.7.9 and OpenSSL 1.0.2

[anentropic]

There seems to be a problem between Python 2.7.9, OpenSSL 1.0.2 installed via Homebrew

Some details here:
docker/docker-py#465

I don't know what the underlying problem is but it looks like it might be something that could be solved in Homebrew

[DomT4]

Does it fix anything if downstream upgrade their version of Requests? It should be done anyway - The downstream mandated version of that library has a known security vulnerability.

[anentropic]

a better related issue link here: docker/compose#890

it seems both docker-py and docker-compose pin their requests version to 2.5.x

[DomT4]

@tdsmith is the magician of all things Python, but as far as I know, since upgrading to Requests 2.6.0 we haven't had any reports of OpenSSL breakage against it - If that indeed proves to be the problem here. M2Crypto has some known failures, but downstream doesn't seem to depend on that at all.

[tdsmith]

The idea that packages required for SNI are missing sounds true to me: docker/docker-py#465 (comment)

Installing them with pip with Homebrew python active won't work because Homebrew's docker-compose formula uses system Python. They should be added as resources to the formula. This looks like an issue with pip installing docker-compose while using python 2.7.9 and a modern OpenSSL.

Thanks for filing against Homebrew. It sucks that this was a (our?) problem for two months and nobody's told us.

Can you give me instructions for reproducing, starting just after a fresh install of OS X on a new machine? I don't know how to use anything in the Docker ecosystem.

[tdsmith]

Though people are saying that the Homebrew formula just works, probably because it's using system Python and system OpenSSL, so if installing the extra packages requests needs for SNI works (in a venv, say), it's probably a case of docker underspeccing the install-requires packages: docker/compose#890 (comment)

[anentropic]

I believe my problem started when I upgraded to Py 2.7.9 via Homebrew

I already had fig (now called docker-compose) install via pip in a virtualenv

After the Python upgrade this stopped working. I think I deleted and recreated my virtualenv using the new Python, but didn't help.

I was able to make docker-compose work by removing it from my virtualenv and installing via Homebrew, though I still had problems with some tools which used docker-py.

Disabling TLS in my boot2docker vm got things working again, but just as a workaround I guess.

[tdsmith]

Can you give me a quick bulleted instruction list on how to set up boot2docker and whatever else I need to do in order to see this breaking?

[anentropic]

with brew python 2.7.9 and OpenSSL 1.0.2 installed, in a test directory

create the following file as docker-compose.yml:

test:
  image: ubuntu

then:

$ brew install boot2docker
$ boot2docker init
$ boot2docker up
$ $(boot2docker shellinit)
$ mkvirtualenv docker
(docker)$ pip install docker-compose
(docker)$ docker-compose up
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

and

(docker)$ pip uninstall docker-compose
(docker)$ deactivate
$ brew install docker-compose
$ workon docker
(docker)$ docker-compose up
Recreating docker_test_1...
Attaching to docker_test_1
docker_test_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

[virtuald]

Any progress on this? It's pretty easy to reproduce, and the instructions @anentropic posted appear to be good enough to do it.

[erickt]

I just did a brew reinstall openssl python and it seemed to clear it up for me. Does that fix things for anyone else?

[bikeNomad]

I had a failure in make test in the openssl installation; after I did
brew uninstall --force openssl python
both installations worked fine.

[kyounger]

Still an issue here, uninstall/reinstall doesn't work.

[DomT4]

The above is unrelated by the looks of it. I suspect that may be a boxen exclusive issue. A full brew gist-logs openssl will help more than the screenshot though.

[dcousineau]

I believe this issue is very related to this persistent pain in my butt: GoogleCloudPlatform/gcloud#25

Unfortunately I don't have detailed notes but last time this hit me I traced it down to SSL connection issues (I can't remember if requests was involved)

[kyounger]

I had an xcode command line tools update that I needed to apply, and then the brew reinstall openssl python worked.

[TimJDFletcher]

Confirmed that updating xcode and reinstalling python and openssl from brew has fixed this for me.

Has anyone tried the hint given during openssl brew install of running: /usr/local/opt/openssl/bin/c_rehash

[dcousineau]

I was not able to have this work reinstalling openssl, python, and running the c_rehash :/

[TimJDFletcher]

It worked for me on fully updated MacOSX 10.10 and Xcode, which version of MacOSX are you on?

[dcousineau]

The machine I ran it on was fully updated (Xcode) 10.10.2.

I'm going to carve out some time today to try this out on my machine which is 10.10.4 (public beta) fully updated as well as try the possible solution listed in tethysplatform/tethys#63

I'll report back later

[aanand]

I'm also investigating this over on docker/compose#890.

Relevant debug info (would be good to know your output for this, @dcousineau and @TimJDFletcher):

$ python -V
Python 2.7.10
$ python -c 'import ssl; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.2a 19 Mar 2015

[aanand]

Also, forcing an older OpenSSL doesn't work for me - it causes Python to error out on import ssl:

$ brew install https://raw.githubusercontent.com/Homebrew/homebrew-versions/master/openssl101.rb

$ brew switch openssl 1.0.1j
Cleaning /usr/local/Cellar/openssl/1.0.1f
Cleaning /usr/local/Cellar/openssl/1.0.1j
Cleaning /usr/local/Cellar/openssl/1.0.2a-1
Opt link created for /usr/local/Cellar/openssl/1.0.1j

$ which python
/usr/local/bin/python

$ python -V
Python 2.7.10

$ python -c 'import ssl; print ssl.OPENSSL_VERSION'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 97, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: dlopen(/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so, 2): Symbol not found: _SSL_CTX_set_alpn_protos
  Referenced from: /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so
  Expected in: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
 in /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ssl.so

[munhitsu]

@aanand at the moment my winner on OSX (10.10.3) is to roll back to native python. It's fresh enough for dev and there are no issues with SSL.
I've tried fresher requests lib while on brew based python 2.7.10 - failed
fresher urllib3 lib while on brew based python 2.7.10 - failed

$ which python
/usr/bin/python

$ python -V
Python 2.7.6

$ python -c 'import ssl; print ssl.OPENSSL_VERSION'
OpenSSL 0.9.8zd 8 Jan 2015

GIST: https://gist.github.com/munhitsu/ba11f4728d726e7cf254

[dcousineau]

@munhitsu that's what I'm having to do currently.

[TimJDFletcher]

I get this version from python:

dhcp-1-122:~ tim$ python -V
Python 2.7.9

dhcp-1-122:~ tim$ python -c 'import ssl; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.2a 19 Mar 2015

[tdsmith]

After looking at this I do not expect that this is a Homebrew bug or that it can or should be fixed in Homebrew, so I'm closing this ticket. Please open a new ticket for any issues that do not involve receiving a SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) message when running docker-compose installed by pip.

[tdsmith]

The workaround I recommend is running pip install requests[security] in whatever context you plan to run docker-compose from. This works by building cryptography against the old system OpenSSL.

[PavelPolyakov]

@tdsmith

The workaround I recommend is running pip install requests[security] in whatever context you plan to run docker-compose from. This works by building cryptography against the old system OpenSSL.

Do you mean just run it in the command line? Or it should be something more intelligent.
I have tried to run it, simply being in the folder with docker files, and it had no effect, maybe you can share some more information?

Regards,

[tdsmith]

That is no longer necessary and I do not recommend it; you will not experience this issue with an up-to-date version of boot2docker or if you use docker-machine.

[PavelPolyakov]

@tdsmith

I have tried several variants:

1. install everything via the pkg file from the docker website
2. install everything via brew - each time I have the SSL issue

docker-machine has the latest version

Would you recommend some special variant of the installation?

[tdsmith]

I have no idea what problem you're experiencing, sorry. I think the root cause must be different.