PEP 466: backport ssl changes #65507
Comments
|
I'm interested to assist with all back port tickets as soon as my internet connection is fixed. A technician is going to check my line again on Tuesday. |
|
I'm not really interested to assist with backport tickets myself. You may nosy me but I may not care at all :) |
|
Christian, Alex, Donald - is there an in-progress patch for this available at all? 2.7.7rc1 is bearing down on us quite rapidly. Given the likely magnitude of this change, should we consider targeting 2.7.8, and give the OpenStack CI team and others a greater chance to check for any issues based on a source checkout before we publish it for general consumption? |
|
I have not started on this patch yet, I'd wanted to get the other, smaller, backports done first. Unless someone else is planning on getting to this, I think 2.7.8 is more realistic. |
|
I agree that 2.7.8 is more realistic. This particular backport is going to be the most work due to the scope of it. |
|
2.7.8 will likely be earlier than expected in order to address the latest OpenSSL update for the Windows installers. So while the likely time frame for this hasn't changed (i.e. November'ish 2014), that release is now expected to be 2.7.9 (assuming the OpenSSL review doesn't find any more surprises, which is a big assumption). |
|
On 20.06.2014 09:26, Nick Coghlan wrote:
I think we need to be more careful about using those patch level release IMO, now would be a good time to discuss how we should deal with |
|
I just wanted to note that I've been actively working on this, but it's being difficult in ways I hadn't predicted :-) Will send an update to python-dev in the next week or so. |
|
MAL - agreed on the version numbering implications of treating OpenSSL CVE's as CPython CVE's, but I think Guido pretty much answered that when he extended the 2.7 EOL to 2020 (i.e. we were going to hit 2.7.10 within the next couple of years regardless). Starting a python-dev thread on that topic in order to reach a broader audience is still a reasonable idea, though. |
|
On 20.06.2014 16:38, Nick Coghlan wrote:
Done. |
|
The attached patch (drafted by myself, and David Reid) backports all of the SSL module (and tests!!!) to Python 2.7. All tests pass on my machine (OS X 10.9), I haven't tested against other platforms. I /suspect/ the best way to review this patch will be by looking at the diff between this patch, and the master branch, in the Lib/ssl.py, Doc/Library/ssl.html, and Modules/_ssl.c files, but of course feel free to review however you like :-) You can see the complete branch history at: https://github.com/alex/cpython/commits/backport-ssl For this to work you must apply the patch from http://bugs.python.org/issue22023 first. |
alex
added
extension-modules
|
New version of this patch fixes a bunch of versionadded and changeds in the docs that referred to the wrong version. |
|
Try #3: This fixes a bug with NPN (I hadn't tested with an OpenSSL that supports NPN, I have now), and cherry-picks the typo fixes from 967311e6c0d2 |
|
Awesome! :) I'll try to find some time to check your work when I'm back from EuroPython. |
|
Thanks for working through this! I've kicked it in the direction of the Fedora Python SIG folks (https://lists.fedoraproject.org/pipermail/python-devel/2014-July/000611.html), since it would be good if we could get it tested before it makes its way into an upstream release. |
|
New patch cherry pick's the fix from bpo-22074. |
|
Some results from a quick build/test on OS X 10.9.4 with Xcode 5.1.1 clang and openssl 1.0.1h from MacPorts (similar results when built with Apple-supplied openssl 0.9.8y and on OS X 10.5.8 ppc with Apple-supplied openssl 0.9.7l):
./python -m test.regrtest -v -uall,-largefile test_ssl
/python -m test.regrtest -v -uall test_poplib
== CPython 2.7.8+ (default, Jul 26 2014, 23:44:13) [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
== Darwin-13.3.0-x86_64-i386-64bit little-endian
== /py/dev/27/source/build/test_python_33661
Testing with flags: sys.flags(debug=0, py3k_warning=0, division_warning=0, division_new=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, tabcheck=0, verbose=0, unicode=0, bytes_warning=0, hash_randomization=0)
test_poplib
test_dele (test.test_poplib.TestPOP3Class) ... ok
test_exceptions (test.test_poplib.TestPOP3Class) ... ok
test_getwelcome (test.test_poplib.TestPOP3Class) ... ok
test_list (test.test_poplib.TestPOP3Class) ... ok
test_noop (test.test_poplib.TestPOP3Class) ... ok
test_pass_ (test.test_poplib.TestPOP3Class) ... ok
test_retr (test.test_poplib.TestPOP3Class) ... ok
test_rpop (test.test_poplib.TestPOP3Class) ... ok
test_stat (test.test_poplib.TestPOP3Class) ... ok
test_top (test.test_poplib.TestPOP3Class) ... ok
test_uidl (test.test_poplib.TestPOP3Class) ... ok
test_user (test.test_poplib.TestPOP3Class) ... ok
testTimeoutDefault (test.test_poplib.TestTimeouts) ... ok
testTimeoutNone (test.test_poplib.TestTimeouts) ... ok
testTimeoutValue (test.test_poplib.TestTimeouts) ... ok
test__all__ (test.test_poplib.TestPOP3_SSLClass) ... Exception in thread Thread-16:
Traceback (most recent call last):
File "/py/dev/27/source/Lib/threading.py", line 810, in __bootstrap_inner
self.run()
File "/py/dev/27/source/Lib/test/test_poplib.py", line 132, in run
asyncore.loop(timeout=0.1, count=1)
File "/py/dev/27/source/Lib/asyncore.py", line 220, in loop
poll_fun(timeout, map)
File "/py/dev/27/source/Lib/asyncore.py", line 156, in poll
read(obj)
File "/py/dev/27/source/Lib/asyncore.py", line 87, in read
obj.handle_error()
File "/py/dev/27/source/Lib/asyncore.py", line 83, in read
obj.handle_read_event()
File "/py/dev/27/source/Lib/asyncore.py", line 443, in handle_read_event
self.handle_accept()
File "/py/dev/27/source/Lib/test/test_poplib.py", line 143, in handle_accept
self.handler = self.handler(conn)
File "/py/dev/27/source/Lib/test/test_poplib.py", line 242, in __init__
self.push('+OK dummy pop3 server ready.')
File "/py/dev/27/source/Lib/test/test_poplib.py", line 63, in push
asynchat.async_chat.push(self, data + '\r\n')
File "/py/dev/27/source/Lib/asynchat.py", line 193, in push
self.initiate_send()
File "/py/dev/27/source/Lib/asynchat.py", line 244, in initiate_send
self.handle_error()
File "/py/dev/27/source/Lib/asynchat.py", line 242, in initiate_send
num_sent = self.send(data)
File "/py/dev/27/source/Lib/asyncore.py", line 374, in send
result = self.socket.send(data)
File "/py/dev/27/source/Lib/ssl.py", line 662, in send
return self._sslobj.write(data)
SSLWantReadError: The operation did not complete (read) (_ssl.c:1611)^CWarning -- asyncore.socket_map was modified by test_poplib Test suite interrupted by signal SIGINT.
|
|
Ned, did you apply the patch from http://bugs.python.org/issue22023 first? That bt looks like what you'd see without it. |
|
Does https://github.com/alex/cpython/commits/backport-ssl contain all relevant code changes? I like to review the changes during my train ride home. |
Sorry, missed that. Let's try again with that applied as well: OS X 10.9.4 with Xcode 5.1.1 clang and openssl 1.0.1h from MacPorts
test_ssl: testing with 'OpenSSL 1.0.1h 5 Jun 2014' (1, 0, 1, 8, 15) Traceback (most recent call last):
File "/py/dev/27/root/uxd_macports/lib/python2.7/test/test_ssl.py", line 1475, in test_algorithms
s.connect(remote)
File "/py/dev/27/root/uxd_macports/lib/python2.7/ssl.py", line 810, in connect
self._real_connect(addr, False)
File "/py/dev/27/root/uxd_macports/lib/python2.7/ssl.py", line 801, in _real_connect
self.do_handshake()
File "/py/dev/27/root/uxd_macports/lib/python2.7/ssl.py", line 774, in do_handshake
self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)Ran 99 tests in 13.035s FAILED (errors=1, skipped=4) when built with Apple-supplied openssl 0.9.8y on OS X 10.9.4:
test_ssl: testing with 'OpenSSL 0.9.8y 5 Feb 2013' (0, 9, 8, 25, 15)
test_nonblocking_send (test.test_ssl.ThreadedTests) ... server: new connection from ('127.0.0.1', 50583)
server: connection cipher is now ('AES256-SHA', 'TLSv1/SSLv3', 256)
server: selected protocol is now None
Exception in thread Thread-36:
Traceback (most recent call last):
File "/py/dev/27/root/uxd/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/py/dev/27/root/uxd/lib/python2.7/test/test_ssl.py", line 1636, in run
self.write(msg.lower())
File "/py/dev/27/root/uxd/lib/python2.7/test/test_ssl.py", line 1581, in write
return self.sslconn.write(bytes)
File "/py/dev/27/root/uxd/lib/python2.7/ssl.py", line 622, in write
return self._sslobj.write(data)
error: [Errno 32] Broken pipeok on OS X 10.5.8 ppc with Apple-supplied openssl 0.9.7l:
test_ssl: testing with 'OpenSSL 0.9.7l 28 Sep 2006' (0, 9, 7, 12, 15) Traceback (most recent call last):
File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/test/test_ssl.py", line 2139, in test_rude_shutdown
connector()
File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/test/test_ssl.py", line 2130, in connector
ssl_sock = ssl.wrap_socket(c)
File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/ssl.py", line 868, in wrap_socket
ciphers=ciphers)
File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/ssl.py", line 561, in __init__
self.do_handshake()
File "/Volumes/cache/py/pyb5g4/27/unix/root/lib/python2.7/ssl.py", line 774, in do_handshake
self._sslobj.do_handshake()
error: [Errno 54] Connection reset by peerRan 99 tests in 14.843s FAILED (errors=1, skipped=21)
|
|
Investigations: The test_poplib failures are caused by the backporting of http://bugs.python.org/issue20951, does anyone have opinions on whether we should backport Antoine's fix for poplib, or revert that particular change? I'm not sure how to fix that compilation warning, any suggestions? I've uploaded a new patch which fixes the test_algorithm's failure with non-system OS X. |
|
Latest patch makes the needed change to Makefile.pre.in |
|
Would it be possible to also backport the changes to httplib.py to support TLS SNI as many libraries (including setuptools) relies on this module to download files and some servers reject clients not supporting TLS SNI (such as cloud.github.com)? I have added a patch for Python 2.7.8. Thanks for working on the backports of SSL changes! |
|
Hi Arnaud, such a change is out of scope for PEP-466. To minimise the scale of the changes in a maintenance release, it was deliberately limited to provide the core infrastructure, so at least third party libraries like requests can support SNI in Python 2, without touching every networking module in the Python 2 standard library. However, you're welcome to write a follow up PEP suggesting expanding the scope of the changes out to at least offering SNI support in other standard library modules. While I can't promise such a proposal would be approved, I do agree the question is worth asking. The risk of that change can then be independently assessed vs advising users that need SNI support in Python 2 to install a third party module like requests. (The bootstrapping problem is likely to be addressed by backporting the relevant parts of PEP-453, although that project too is in need of a volunteer to write the enhancement proposal). |
|
I think we probably want to revert that particular change. Afaik it wasn't added to 3.4 because of the danger of breaking things so we probably shouldn't add it to 2.7. |
|
Latest patch does as Donald suggests and rolls back the WantWrite changes, fixing poplib. |
|
To help keep the file list clear, I unlinked the older versions of the patch. They're still available in the issue history below. Alex, could you generate a version using the hg diff format (so the review gets generated automatically), or else add it manually to Rietveld? |
|
I've been doing this work in a git repo, do you know how to generate an hg-formated diff with git? |
|
Doesn't hg accept git diffs? On Wed, Aug 6, 2014, at 21:31, Alex Gaynor wrote:
|
|
The problem is that the Rietveld integration isn't able to figure out the base revision, so it isn't generating the review automatically. Times like this it would be nice to be able to specify a "base branch" explicitly... In other news... the outcome of the Fedora thread is that Slavek is just going to go ahead and apply the patch to Python 2.7 RPMs in Fedora Rawhide next week. That will hopefully be entirely uneventful, but if it isn't, at least we'll know *before* merging it upstream :) |
|
Le 07/08/2014 04:34, Nick Coghlan a écrit :
I doubt adding a ton of new APIs and code can be uneventful, but good |
|
New patch should be in the "mercurial" diff format. |
|
On 7 Aug 2014 23:11, "Antoine Pitrou" <report@bugs.python.org> wrote:
They don't call it "Rawhide" for nothing! :) |
|
Hi everyone, I went ahead and I've applied ssl-backport.diff (alex, 2014-08-07 18:49) patch into Python 2.7.8 on Fedora Rawhide (currently only scratch build). My report: Next issue was/is distro specific, there is disabled SSLv2 protocol when SSLv23_method is used[0]. However python3 builds fine[1]: In test_ssl.py:test_protocol_sslv23 when calling In python 2 this will raise a socket.error[2] instead, but test_protocol_sslv23 is catching ssl.SSLError, shouldn't be that changed to socket.error so both python2 and python3 behave the same? [0] SSLv23 http://pkgs.fedoraproject.org/cgit/openssl.git/tree/openssl-1.0.1h-disable-sslv2v3.patch |
|
Thanks for testing this out Robert! The attached patch fixes the error you saw. |
|
New patch additionally backports the Tools/ssl/ directory from Python3, which has two utilities for generating some of the code used. Thanks to Benjamin for catching this. |
|
Last version had some stray stuff due to the fact that I don't know how to use version control. New version should resolve that. |
|
Running test_ssl gives this: % ./python Lib/test/regrtest.py test_ssl
test_ssl
Exception in thread Thread-104:
Traceback (most recent call last):
File "/home/benjamin/dev/python/2.7/Lib/threading.py", line 810, in __bootstrap_inner
self.run()
File "/home/benjamin/dev/python/2.7/Lib/test/test_ssl.py", line 1594, in run
if not self.wrap_conn():
File "/home/benjamin/dev/python/2.7/Lib/test/test_ssl.py", line 1545, in wrap_conn
self.sock, server_side=True)
File "/home/benjamin/dev/python/2.7/Lib/ssl.py", line 350, in wrap_socket
_context=self)
File "/home/benjamin/dev/python/2.7/Lib/ssl.py", line 561, in __init__
self.do_handshake()
File "/home/benjamin/dev/python/2.7/Lib/ssl.py", line 784, in do_handshake
self._sslobj.do_handshake()
error: [Errno 104] Connection reset by peer |
|
On what platform? |
|
Gentoo Linux. Openssl 1.0.1i On Tue, Aug 19, 2014, at 19:29, Alex Gaynor wrote:
|
|
In load_cert_chain, you may find it convenient to not Py_DECREF and Py_CLEAR keyfile_bytes and certfile_bytes, which aren't PyObject *. |
|
Latest patch fixes both the issues Benjamin noted. |
|
I spent hours looking at this patch, which certainly doesn't constitute a real review, but is probably about as good as your going to get on this behemouth. Anyway, Alex knows he's on the hook for when things start going sour. |
|
New changeset 221a1f9155e2 by Benjamin Peterson in branch '2.7': |
|
The current implementation doesn't work with Unicode file paths. Try passing a Unicode string e.g. as "cafile" into context.load_verify_locations(). It calls PyString_AsEncodedObject() on it, which then fails with a PyErr_BadArgument() on the entry type check. |
|
Thanks for the report, I've filed: http://bugs.python.org/issue22244 to track that issue. |
|
Thanks for getting this in, folks! As Alex suggests, we can file any identified regressions as new issues. |
|
Note that this change broke eventlet: eventlet/eventlet#135 |
|
Filed http://bugs.python.org/issue22438 to track it. |
|
causing bpo-22523, still referencing _ssl.sslwrap. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: