TLSv1.0/TLSv1.1 rolling brownouts are not transparent

[awallace-cray]

FYI, The pip clients we're using sure don't pass along any error 403 message.
The connection "succeeds" but the client just won't find anything it is looking for (pip install).
No response needed.
Thanks.

[alex]

What version of pip are you using?

[awallace-cray]

Looks like 6.0.6.
It is baked-in.

[alex]

Baked into what?

Can you try with the latest pip, 9.0.3?

[awallace-cray]

Baked-in the project config files. Which are probably up for grabs now anyway, because the TLSv1.0/TLSv1.1 change.
Will let you know about trying pip 9.0.3.
I think root cause on our side, is using old SLES-11 machines in this project.

[dstufft]

Ancient versions of OpenSSL will definitely be an issue.

[alex]

I don't know anything about SLES, but https://www.suse.com/documentation/suse-best-practices/singlehtml/securitymodule/securitymodule.html seems like the relevant doc

[awallace-cray]

Roger that. Will need new bits. Will take time. Meanwhile, a local mirror or cache or such, to keep project going. Tearing hair out today.

[ncoghlan]

Using devpi-on-recent-Python as a caching proxy could be a pretty robust workaround for older pip clients. I'm not sure we could provide that as a generic pre-assembled solution (since the specifics will differ depending on the exact constraints preventing a pip upgrade), but I believe it would let devpi handle the TLS v1.2 link to PyPI, while talking whatever the legacy client needs on the local side of things (for a local host loopback connection, it could even be HTTP, without using TLS at all).

[awallace-cray]

Our local PyPI mirror supports http and is working well. I did not build this mirror, and do not know how it works.

Pip install needs parameters --index-url $mirror_url --trusted-host $mirror_hostname,
which can be passed to pip in different ways. All good.

However, many of our scripts also use easy_install. To go get virtualenv, for example.
Easy_install works w our mirror if you add -i $mirror_url to its command line.
I wish I had an "easier" way to pass easy_install parameters, like with conf files or an env variables. Ideally, this should work even with old versions of easy_install.

[alex]

We strongly recommend you stop using easy_install; it's been deprecated for a long time and hasn't really kept up with improvements to the packaging ecosystem.

[awallace-cray]

Good advice.

Do you happen to know when the TLSv1.0/TLSv1.1 policy will be fully enforced on pypi.org?

• Thanks

[alex]

We'll be gradually ratcheting up the brownout periods over the next few weeks; April 8th is our current target for fully disabling them.

[awallace-cray]

Said I'd get back to you on the pip error message.

time: 2018-03-23 08:06 PT
host: an old SLES-11 machine, but I am using a recently-built Python 2.7 (user-built from source)

$ pip install argparse # No 403 error message
Collecting argparse
  Could not find a version that satisfies the requirement argparse (from versions: )
No matching distribution found for argparse

$ pip install -v argparse # there it is
Collecting argparse
  1 location(s) to search for versions of argparse:
  * https://pypi.python.org/simple/argparse/
  Getting page https://pypi.python.org/simple/argparse/
  Looking up "https://pypi.python.org/simple/argparse/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/argparse/ HTTP/1.1" 403 109
  Status code 403 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/argparse/: 403 Client Error: Brownout of Legacy TLS for url: https://pypi.python.org/simple/argparse/ - skipping
  Could not find a version that satisfies the requirement argparse (from versions: )
Cleaning up...
No matching distribution found for argparse

$ pip --version
pip 9.0.1 from /var/tmp/chapelu/awallace/venv/lib/python2.7/site-packages (python 2.7)

$ python --version
Python 2.7.13

$ type python
python is hashed (/ptmp/chapelu/awallace/venv/bin/python)

[alex]

@dstufft this seems like an issue, pip should, without -v, recognize a non-200 status code and print something useful. Is there an existing pip tracking bug for this?

[dstufft]

It did print the status code out right?

Could not fetch URL https://pypi.python.org/simple/argparse/: 403 Client Error: Brownout of Legacy TLS for url: https://pypi.python.org/simple/argparse/ - skipping

[dstufft]

I don't tihnk there's a bug for tracking any additional changes to that though, no.

[alex]

It only printed that with -v though -- it should do more by default.

[dstufft]

Oh right. Yea open an issue

[alex]

pypa/pip#5111

[jvanasco]

FYI, pip 9.0.1 does not pass on any message or note a 403. It just says no matching versions are found, for everything. One must pass in -vvvv to see what happened.

[ntwaddell]

So what needs to be done to fix this? it's causing some of my python projects to fail in TravisCI

Could not fetch URL https://pypi.python.org/simple/pip/: 403 Client Error: Brownout of Legacy TLS

Just upgrade OpenSSL?

[jvanasco]

@ntwaddell try to upgrade pip first. you can download/run the get-pip.py from here: https://bootstrap.pypa.io/

that should fix several environments.

[alex]

What platform are you hitting this on with Travis CI? We should make sure any builtin packages they have are updated.

[ntwaddell]

@alex I'm using TravisCI Enterprise, it always takes them longer to update their enterprise platform.

Build language: python
Build group: stable
Build dist: precise
Description: Ubuntu 12.04.5 LTS
Release: 12.04

OpenSSL is OpenSSL 1.0.1 14 Mar 2012

[alex]

@ntwaddell On Ubuntu Precise you should be able to get a working OpenSSL by doing sudo apt-get update && sudo apt-get install --only-upgrade openssl libssl-dev.

[jvanasco]

@ntwaddell FYI, that's the first heartbleed release. That should have been upgraded years ago.

[gajjar8055]

I have spent around 5 hours and almost die, man

[shawngustaw]

I'm not sure if this is related at all but a coworker was having similar issues and brew install python@2 ended up fixing it. Oddly, downloading Python 2.7.14 directly from the Python website didn't work, but installing via brew and overwriting seems to be using the correct OpenSSL version.

[awallace-cray]

I can confirm: brew install python@2 installs newer version of easy_install and pip in /usr/local/bin, and these clients can connect to PyPI even when the system-installed Python-2 easy_install or pip cannot.

I've read posts about brew install python fixing the same or similar problem for Python 3.

[brainwane]

A comment on LWN suggests we should have an announcement about this on python.org. Someone could write something to post to http://blog.python.org/ which syndicates to that front page -- caution that only about the first ~30 characters will show up.

[alex]

At this point we've had TLS 1.0/1.1 disabled 100% of the time for a few days with limited feedback, I'm reluctant to think we need further publication of this.

[damienrj]

I know this caught my company by surprise and has broken a few things.

[alex]

Can you suggest what communications channels we could have promoted the brownouts on to help you see them?

[brainwane]

Now that we have entirely stopped supporting TLS 1.0 and 1.1, I am closing this issue. To be notified of future breaking changes to PyPI, please subscribe to the pypi-announce mailing list.