Clean install through pip fails with SSL/TLS issues

[robinsdg]

As recommended (and this should likely be in the README), I’m using a virtual environment. Running the command pip install --extra-index-url https://pypi.org --index-url https://test.pypi.org/simple afdko (even with the --trusted-host flags), I however get an error that I can’t trace back reliably – I think I’m out of my depth on this one. I’ve pasted in the error I get. This is on macOS Sierra, relatively clean machine and, as said, in a new virtual environment for the express purpose of testing the new FDK.

It isn’t a firewall issue as far as I can tell, as this happens both on our corporate network and from my tethered phone on 4G.

  Could not fetch URL https://test.pypi.org/simple/afdko/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping
  Could not find a version that satisfies the requirement afdko (from versions: )
No matching distribution found for afdko

[anthrotype]

what python are you using?
what's the output of which python or python --version?

side note, users shouldn't rely on the presence of a package on test.pypi.org; that's only temporary location for the developers to test the packaging tools.

[robinsdg]

I’m using the recommended 2.7.x. I understand the developer caveat – I just really need the FDK update (in particular to checkOutlinesUFO). Regardless, pip finds no evidence of any afdko package. It might be my local settings, but, this is how macOS Sierra ships – then I won’t be the only one with this problem.

(fdk-test) 
robin.mientjes in ~/Code $ which python
/Users/robin.mientjes/Code/fdk-test/bin/python
(fdk-test) 
robin.mientjes in ~/Code $ python --version
Python 2.7.14

[anthrotype]

it's probably this bug:
pypa/pip#829 (comment)

People with old versions of openssl that don't support sha1 SSL certificates need to upgrade or else they are insecure.

[anthrotype]

how did you install python then?

[robinsdg]

A valid question. From the site, I think. You’re probably right about openssl, I’m trying to update that now – mixed results so far but I’ll update you. When I figure it out, might be a good item for the README/setup flow.

[anthrotype]

users shouldn't need to mess with openssl and such. You just need a fairly recent version of python 2.7 (that links with the up-to-date openssl).
The important thing to understand is whether you're using the OSX pre-installed python distribution (ie. /usr/bin/python) or any other one (e.g. official binaries from Python.org, homebrew, macports, pyenv, etc.). Then we can debug this further.

[anthrotype]

also, check what version of pip you have, ensure you're running the latest.

[anthrotype]

and try running python -c "import ssl; print ssl.OPENSSL_VERSION"

[robinsdg]

Python is the installer from python.org. pip is 9.0.1. The openssl import gives me OpenSSL 0.9.8zh 14 Jan 2016 which seems to be the outdated one.

[anthrotype]

that is weird.

[anthrotype]

so you downloaded and installed the 2.7.14 python.org distribution, with default settings. When you do which python in the Terminal (outside of any virtualenv), what do you get?

[robinsdg]

/Library/Frameworks/Python.framework/Versions/2.7/bin/python

[anthrotype]

I just tried and am also seeing that. Investigating

[anthrotype]

sorry it looks like a bug... https://bugs.python.org/issue17128

well, one way for you to fix this would be to use homebrew to install python2. Note that if you do that you have to call python as python2 (or python3).

Homebrew links it with the most recent openssl, for sure.

[anthrotype]

this is explained in the README of the mac python.org installers, if we care to read:

For OS X 10.6+, Apple also provides OpenSSL 0.9.8 libraries. Apple's 0.9.8 version includes an important additional feature: if a certificate cannot be verified using the manually administered certificates in /System/Library/OpenSSL, the certificates managed by the system security framework In the user and system keychains are also consulted (using Apple private APIs). For this reason, for 2.7.9 the 64-bit/32-bit 10.6+ python.org variant continues to be dynamically linked with Apple's OpenSSL 0.9.8 since it was felt that the loss of the system-provided certificates and management tools outweighs the additional security features provided by newer versions of OpenSSL. This will likely change in future releases of the python.org installers as Apple has deprecated use of the system-supplied OpenSSL libraries. If you do need features from newer versions of OpenSSL, there are third-party OpenSSL wrapper packages available through PyPI.
The bundled pip included with 2.7.9 has its own default certificate store for verifying download connections.

[robinsdg]

Homebrew links it with the most recent openssl, for sure.

I really wish it would. Still fighting. I’m kinda lost here. I can’t imagine this problem will remain exclusive to me.

[anthrotype]

doesn't it work?

make sure you're actually calling the correct python2 and pip exectutables.
call homebrew's pip with python2 -m pip ...

[robinsdg]

Well, outrageously, the Homebrew Python 2 install didn’t set itself up on the PATH. I can’t figure out why or how, and frankly this is as far as I’ll go today with this bizarre set of issues. I downloaded the .whl file and installed it locally. Now I’m back to the comfort of makeOTF reporting an error I don’t understand (src glyph aring is newer than processed layer. programName: CheckOutlines).

[robinsdg]

Okay, the stated command in the README works from python2 -m …, which is great. I just can’t tell you whether I understand how I managed this, and I don’t look forward to learning about which parts I broke doing this. Now it’s just back to font issues again.

[anthrotype]

Sorry about that, this is a bit out of our control...
I have commented on the open issue17128 on python.org bug tracker.
Thanks for helping me debug 👍

[robinsdg]

Thanks for debugging. One last question: the README says pip will override the AFDKO on the path – but checkOutlinesUFO still runs 1.22, not 2.0. But the source file that it refers to definitely says 2.0. It points me to the virtualenv version, which points to the AFDKO SharedData, which contains only 2.0. Or am I going mad?

[anthrotype]

sorry, I was just passing by and got curious with the error. I'm not a afdko developer myself, and don't know how this was set up... good luck! 😁

[anthrotype]

this just worked for me (using homebrew python2 on macOS High Sierra):

$ python2 -m virtualenv afdko-test
$ source afdko-test/bin/activate
$ pip install --extra-index-url https://pypi.org/simple --index-url https://test.pypi.org/simple afdko
$ which checkOutlinesUFO
/Users/clupo/afdko-test/bin/checkOutlinesUFO
$ checkOutlinesUFO --version
checkOutlinesUFO 2.0.0

[robinsdg]

Yeah, I get stuck at pip install following this. Then I get the SSL/TLS issue. If I try to link/relink the Homebrew python2, it still keeps referring to the Python.org-installer version. What do I do there?

[anthrotype]

try uninstalling and reinstalling homebrew python2.

the /usr/local/bin/python2 has to be a symlink to /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/bin/python2.7

don't do it yourself, let homebrew do it.

[anthrotype]

when you are outside a virtual environment, you call pip with the python interpreter followed by -m, e.g. python2 -m pip ...

This is to be sure you are using the pip executables that is linked to that specific python interpreter.

only when inside a virtualenv you can unconditionally do pip install ... without worrying that it's the correct pip. It must be if virtualenv is working, as it prepends its own bin subfolder to the $PATH for the current shell only (until you deactivate).

[robinsdg]

I uninstalled and reinstalled, it says it’s pointing to the Python2 in /Cellar/, now it’s lost all knowledge of virtualenv and pip. That’s the brew version, apparently.

[robinsdg]

After easy_installing virtualenv on the Homebrew Python2, it seems to now work. I have to aggressively prefix python2 -m to everything inside the virtual environment, but now I get the new files to work.

[anthrotype]

let's keep this issue open until we figure out a workaround for python.org installs or improve the docs.

don't ever use easy_install, it's deprecated. You can and should use pip for everything.

you don't need to call python as python2 from within the virtualenv. If you sourced the virtual environment properly, then you just call pyhon and that will mean the current virtualenv's python executable, whatever that is.

[anthrotype]

to recap

1. brew install python2
2. python2 -m pip install virtualenv
3. python2 -m virtualenv afdko-whatever-venv-name
4. source afdko-whatever-venv-name/bin/activate

now we are inside, for as long as we do deactivate. any calls to python or pip will be forwarded to the currently active virtual environment. installing stuff will go to the site-packages folder of this current virtual environment only, without polluting the global site.

All this above must work, if it doesn't we have a problem.

[robinsdg]

From scratch, this process does indeed work. Thanks a lot! We should probably recommend this process for any new installs.

[anthrotype]

still, it's a pity that the afdko package be tied with any particular python distribution. The advantage of moving to a setup.py kind of installation was to free oneself from having to distribute its own python installation, and be treated like any other python library distributed via PyPI.

I think the issue can be mitigated by the fact that the SSLError above with the official Python.org binary only happens when attempting to install from the test.pypi.org server. It doesn't happen when one pip installs from the regular pypi, does it?

If so, then this is just a temporary issue which will go away once afdko is uploaded to the actual PyPI server, instead of the test one.

[robinsdg]

In my search for answers to these issues, I found plenty of similar complaints about other packages. pypi.org is https, so it’s very well possible that this problem isn’t going away when we stop with the test server.

[anthrotype]

Also, note that none of the instructions I gave above is anything special or related to the afdko or homebrew (well, of course, besides invoking brew itself).
This is all standard python and info is available on https://packaging.python.org/ so I don't think it should be explained in the AFDKO or any other python package's documentation, besides having a link to the official python docs.

[anthrotype]

it’s very well possible that this problem isn’t going away when we stop with the test server

I just tried to call pip install fonttools from within a virtualenv created with the python.org's python executable (which returns 'OpenSSL 0.9.8zh 14 Jan 2016' as import ssl; print(ssl.OPENSSL_VERSION)) and it worked without problems.

[anthrotype]

they may be using a different certificate format which is unsupported on later OpenSSL versions, on the test server as opposed to the production one.

[robinsdg]

All right, fair enough. Then my hands are off.

[miguelsousa]

it's a pity that the afdko package be tied with any particular python distribution

@anthrotype Is it? I had no idea. What is it tied to? How can we remove that limitation? Sorry, it's not clear from reading the discussion above.

[anthrotype]

I meant that before it used to bundle it’s own python. Let’s not force users to use a particular python now (eg homebrew’s)

[miguelsousa]

Ah, got it.

[pradyunsg]

http://pyfound.blogspot.in/2017/01/time-to-upgrade-your-python-tls-v12.html

It's actually because of lack of TLS 1.2 support on old MacOS System Pythons.

[pradyunsg]

Also, pip 9.0.3 has a patch to use TLS 1.2 on affected systems.

[hadifarnoud]

struggling with is issue too. what is the solution? can't code cuz cant use pip

[readroberts]

See the article 'time-to-upgrade-your-python-tls-v12.html' referenced by pradyunsg above. An alternative is to upgrade your Mac OS to High Sierra; the system python does then support TLS 1.2. It is unfortunate that PyPi and other organizations are being forced to switch over to using TLS 1.2, as it does mean that everyone who wants to use pip with Python 2.7 must either upgrade to High Sierra, or build and install the latest Python distribution. Well, time will make this steadily less of a problem.

[anthrotype]

I think it should be enough to simply upgrade pip to 9.0.3. Did you try that?

[hadifarnoud]

I managed to upgrade it with curl https://bootstrap.pypa.io/get-pip.py | python that fixed the issue.

PS: I am on high sierra.

[simbawesley]

I ran the following commands to resolve the issue

$ curl https://bootstrap.pypa.io/get-pip.py >> get-pip.py
$ python get-pip.py

This upgraded pip to pip-9.0.3 and this version has no issues

[iyurev]

Yes, This's works!!!

[gitanupam]

This could be a result of the recent TLS support deprecation by Python.org sites. Python.org sites have stopped supporting TLS v1.0 and v1.1. Need to upgrade pip (but without using pip) to resolve it. See this SO post with details

[nextkitt]

$ curl https://bootstrap.pypa.io/get-pip.py | sudo python
$ sudo pip install -U virtualenv
$ virtualenv env.test 
$ source env.test/bin/activate
$ pip -V
pip 10.0.1 from xxxx
$ pip install xxx

[readroberts]

Hi nextkitt; I assume you are posting this because installing the AFDKO failed for you. If this is correct, did you also try the last step in gitanumpam's post, of updating setup tools? 'pip install --upgrade setuptools'.

[gfarb]

I know this issue is closed but I had the same issue and with hours of research and attempting potential solutions nothing seemed to work. Ended up uninstalling and reinstalling Python and have not ran into this issue since, I highly suggest you try this.