HTTPS issues with Python 2.

[macfreek]

The current Travis automated tests currently fail for Python 2.6, pypy, as well as Python 2.7 if you are using 2.7.8. or earlier.

The reason seems rather mundane: the tests download the file https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz. This file is not included in the standard distribution due to its size.

HTTPS support in Python 2 is very bad. In this particular case, it tries to download this file using the SSLv3 protocol, which was righteously disabled by github.com after the recent SSL Poodle vulnerability.

Python up to 2.7.8 only seems to try SSLv3, and does not use TLSv1.0. So it fails with an error:

>>> import urllib2
>>> request = urllib2.Request("https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz")
>>> remotefile = urllib.urlopen(request)
urllib2.URLError: <urlopen error [Errno 1] _ssl.c:493: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

Now, this is a bit surprising, as Python 2.6 - 2.7.8 does support TLSv1.0 in the ssl module. However, the urllib2 module (which NBT uses) uses the httplib module, and the httplib module uses the ssl modules, but httplib does not expose this functionality to urllib2 (or to NBT). The situation is actually worse: up to Python 2.7.8, no certificate was ever checked for validity, SNI is not supported, and there are probably more issues if I would really dive into it. [Edit: up to Python 3.3, no certificate was ever checked for validity, see PEP 476].

The situation was actually so bad that the Python core team decided to do the only reasonable thing: they added new features to Python 2.7, and Python 2.7.9 was specifically released to backport the ssl, httplib, urllib2 and ensureip modules from Python 3 to Python 2. The release notes talk about "several significant changes unprecedented in a bugfix release".

Now this is a bit of a pickle: while the NBT code itself should still work, it currently can not be checked on Python 2.6 - 2.7.8. I propose the following:

• As of now, NBT supports Python 2.7.9 and up, and Python 3.2 and up.
• NBT is still compatible with Python 2.6 - 2.7.8, and we like to keep it that way, but this support may be dropped at any time. Basically, as soon as a next issue comes along which takes more than an hour to fix, support is dropped. Supporting older versions just takes too much time. Time we rather spend working on issues like Fix unlink chunk with mismatched lengths status taking minutes #76 or Too many files open in world.iter_nbt (/w fix) #77.
• I'll have a look if we can support Travis testing for 2.6-2.7.8. Either we skip some tests, we rely on an external module, such as request, for tests only, or we look into a quick fix (such as these two: Forcing TLS1+ and monkey-patch ssl.wrap_socket()).

Any feedback, positive and negative, is highly appreciated.

If the feedback is mostly in agreement, I will to create a version 1.4.2 (which should include patches for #76 and #77), with a note that it is the last version that supports Python 2.6 - 2.7.8.

[macfreek]

The patches did not work (I expect because I was modifying ssl parameters, but those changes were local to my script, not to urllib2).

Anyhow, I got requests to work. Indeed great package, much cleaner API.

The bad news, the underlying library is still ssl, which refuses to talk to TLS (any version) for Python 2.6. According to the docs, use PROTOCOL_SSLv23 | OP_NO_SSLv2 | OP_NO_SSLv3, thus 'SSL protocol v2 or v3, but not v2 nor v3'. Right. Apparently, "PROTOCOL_SSLv23" does not mean "protocol v2 or v3", but "the highest protocol that both client and server speak". At least, that's what it means in Python3, and now also in Python 2.7.9. But in Python 2.6-2.7.8, it still means "protocol v2 or v3, but not TLSv1.0, even though TLSv1.0 is supported". So, I can select ssl.PROTOCOL_TLSv1, right? Yes, I can, but only if I use ssl directly. Of course, that only means I have to rewrite both httplib and urllib2.

It starts to daunt on me that there authors of libraries like GnuTLS and NaCl. I hated OpenSSL for it GNU-incompatible licensing (I don't really like GNU for the same reason by the way, but at least they reasoning is correct). I now also start to hate it because it's terrible, terrible interface, and wished Python would have used a sane library.

Maybe I should just drop Python 2.6-2.7.8 and do something fun.

Or just get some sleep.

[macfreek]

If you read this, please reply with the version of Python you use (e.g. my default is 3.3.2 nowadays, though I still use 2.7.9 in some rare cases.).

[macfreek]

OK, the issue was much easier: The ssl module in Python 2.6-2.7.8 does not support HTTPS redirects.

Downloading https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz instead of https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz works perfectly fine.

Apparently, this is also valid for requests.

I'll think a bit what to do -- I'll probably use urllib2 after all, even though I like requests better.

Edit: I was not thinking straight, I was using 2.7.9 instead of 2.7.8. Still doesn't work for 2.7.8 and earlier.

[macfreek]

I wrote a wrapper to call wget to download the file.

I picked wget over curl, because on my laptop, wget was compiled with GnuTLS instead of the dreaded OpenSSL library.

Travis job 89.1 tells the tale:

INFO     Downloading https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz
ERROR    Download https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz failed: <class 'urllib2.URLError'> <urlopen error [Errno 1] _ssl.c:493: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>
ERROR    Download https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz failed: _ssl.c:493: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
INFO     Downloading https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz (with wget)
--2015-01-03 15:33:18--  https://github.com/downloads/twoolie/NBT/Sample_World.tar.gz
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz [following]
--2015-01-03 15:33:19--  https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz
Resolving cloud.github.com (cloud.github.com)... 54.230.68.7, 54.230.71.206, 54.230.68.42, ...
Connecting to cloud.github.com (cloud.github.com)|54.230.68.7|:443... connected.
OpenSSL: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
Unable to establish SSL connection.
('exitcode =', 4)

So urllib2 fails with a sslv3 alert handshake failure. Then, wget -O Sample_World.tar.gz https://cloud.github.com/downloads/twoolie/NBT/Sample_World.tar.gz is used. This also fails with a sslv3 alert handshake failure.

I was surprised.

Just to make it clear: wget is not using any Python at all. 2.6, or anything else. The problem lies in some OpenSSL libraries, and both Python's ssl modules, as well as the wget on the Travis VM (wget 1.3.4 with OpenSSL 1.0.1) don't behave well.

I suspect the following is the root of the problem:

According to the wget 1.12 man page (at the text on --secure-protocol):

[T]he SSL library is given the liberty of choosing the appropriate protocol automatically, which is achieved by sending an SSLv2 greeting and announcing support for SSLv3 and TLSv1.

According to the wget 1.16 man page:

[T]he SSL library is given the liberty of choosing the appropriate protocol automatically, which is achieved by sending a TLSv1 greeting.

Since wget 1.12 fails and wget 1.16 succeeds (although on different machines), this may be the root cause. Or it may not. I no longer care.

I switched from wget to curl for backup-download mechanism and that works on Travis.

Case closed.