Support OpenSSL 1.1.1

[tiran]

BPO	32947
Nosy	@vstinner, @tiran, @benjaminp, @ned-deily, @alex, @dstufft, @stratakis, @hroncok
PRs	bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes #5663 [3.7] bpo-32947: OpenSSL 1.1.1-pre1 / TLS 1.3 fixes (GH-5663) #5923 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-6976) #8760 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-5663) #8761 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 #11612 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 #11612 [3.6] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 #11612 [2.7] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-8761) #11876

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2019-05-29.02:06:57.441>
created_at = <Date 2018-02-25.11:54:40.611>
labels = ['expert-SSL', '3.8', 'type-feature', '3.7']
title = 'Support OpenSSL 1.1.1'
updated_at = <Date 2019-05-29.02:06:57.440>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2019-05-29.02:06:57.440>
actor = 'ned.deily'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2019-05-29.02:06:57.441>
closer = 'ned.deily'
components = ['SSL']
creation = <Date 2018-02-25.11:54:40.611>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 32947
keywords = ['patch']
message_count = 21.0
messages = ['312804', '312897', '317529', '317575', '322490', '322795', '323526', '323901', '323903', '323909', '325793', '325795', '325926', '326719', '326720', '333990', '335623', '342616', '342617', '343853', '343854']
nosy_count = 10.0
nosy_names = ['janssen', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'alex', 'dstufft', 'kroeckx', 'cstratak', 'hroncok']
pr_nums = ['5663', '5923', '8760', '8761', '11612', '11612', '11612', '11876']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32947'
versions = ['Python 3.7', 'Python 3.8']

[tiran]

I'm using this ticket as an epos to track commits and required changes for OpenSSL 1.1.1 and TLS 1.3. Fixes need to be backported to 2.7 and 3.6 to 3.8. We might have to consider backports to 3.4 and 3.5, too.

If all goes to plan, OpenSSL 1.1.1 final is scheduled for 8th May 2018, https://www.openssl.org/policies/releasestrat.html . It will contain support for TLS 1.3. Python should either support TLS 1.3 by then or disable TLS 1.3 by default.

Fixes:

• bpo-20995 added TLS 1.3 cipher suite support
• bpo-29136 added OP_NO_TLSv1_3
• bpo-30622 fixes NPN guard for OpenSSL 1.1.1

Issues:

• A new option OP_ENABLE_MIDDLEBOX_COMPAT is enabled by default. We need to expose the flag to make test pass.
• TLS 1.3 has changed session handling. The current session code cannot handle TLS 1.3 session resumption.
• Threaded echo server and asynchat based tests are failing with TLS 1.3. I haven't analyzed the issue properly. It looks like the server thread dies when a handshake error occurs.

[tiran]

Ned, Benjamin

OpenSSL 1.1.1 is scheduled to be released just before 3.7.0rc1 will come out. I'd rather address as many issues now instead of adding last minute patches to the release candidate.

Once OpenSSL 1.1.1 is out and Python 3.7 supports it, we can discuss backports. 3.6 and 2.7 are going to need fixing, too.

[ned-deily]

Christian, I'm not sure how this issue now differs from bpo-33618 (and whether it can be closed as a duplicate) but, with the delay in OpenSSL 1.1.1 and as discussed over there, full 1.1.1 support will have to wait for 3.7.1 et al so I'm removing the "deferred blocker" status.

[tiran]

3.7 and 3.8 support OpenSSL 1.1.1-pre7-dev. For 3.6 and 2.7 I have to backport some test fixes and documentation.

I prefer to wait until both TLS 1.3 and OpenSSL 1.1.1 have been finalized. Once 1.1.1 is out, I'll fix the outstanding issues on master and then backport all fixes to 3.6 / 2.7.

[hroncok]

Note that Fedora 29 updated openssl from 1.1.0h to 1.1.1-0.pre8 and Python 3.7 tests are failing. Not 100% sure it's related, but full report at:

https://bugzilla.redhat.com/show_bug.cgi?id=1609291

[stratakis]

Yes test_poplib and test_ftplib on fedora rawhide when run against openssl 1.1.1 pre8. Haven't tried the pr7, but assuming that the tests were fine before here is the list of changes between pre7 and pre8:

openssl/openssl@OpenSSL_1_1_1-pre7...OpenSSL_1_1_1-pre8

[tiran]

New changeset 2a4ee8a by Christian Heimes in branch '3.6':
bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-8761)
2a4ee8a

[kroeckx]

This are the errors I'm currently getting testing with the pre9 verion in Debian:
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python2.7/865936/log.gz
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python3.6/865937/log.gz
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python3.7/865938/log.gz

[tiran]

Kurt, can you try again with a current git checkout from master? I fixed a couple of issues lately. CPython master passes all tests with vanilla OpenSSL 1.1.1-pre9. Does Debian change some default settings?

[kroeckx]

This are automated tests for the packages in Debian. I uploaded the pre9 version to unstable, and as a result of that all reverse dependencies got tested. I don't have any experience with python myself.

Anyway, the openssl.cnf in Debian contains:
[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

So you might want to override that during the test suite by calling SSL_CTX_set_min_proto_version(ctx, 0).

[kroeckx]

Christian,

Do you have any update on this? Any idea when we can expect relased python versions that work with OpenSSL 1.1.1?

[tiran]

Soonish, I'm still working on post handshake auth.

[tiran]

Kurt, see bpo-34670 for PHA for server and client side.

[kroeckx]

Do you have any idea when the next release will be? I think python is currently our biggest blocker for getting OpenSSL 1.1.1 in Debian testing.

[tiran]

The release candidates came out a couple of days ago.

[vstinner]

On Fedora 29 with OpenSSL 1.1.1 FIPS 11 Sep 2018, test_connect_cadata() of test_ssl fails randomly:

---

$ ./python -m test -u all -F -m test_connect_cadata test_ssl 
Run tests sequentially
0:00:00 load avg: 0.43 [  1] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/home/vstinner/prog/python/3.6/Lib/test/test_ssl.py", line 1642, in test_connect_cadata
    s.connect(self.server_addr)
  File "/home/vstinner/prog/python/3.6/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/home/vstinner/prog/python/3.6/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/home/vstinner/prog/python/3.6/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/home/vstinner/prog/python/3.6/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

test_ssl failed

== Tests result: FAILURE ==

1 test failed:
test_ssl

Total duration: 131 ms
Tests result: FAILURE
---

This bug has been fixed in master by commit 529525f. It was partially backported in 3.6 with commit 2a4ee8a, but the backport is incomplete.

I wrote PR 11612 to backport remaining fixes.

[vstinner]

New changeset 2149a9a by Victor Stinner (stratakis) in branch '2.7':
[2.7] bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-8761) (GH-11876)
2149a9a

[benjaminp]

Was using OpenSSL to verify hostnames intentionally not backported?

[tiran]

Yes, the feature requires OpenSSL 1.0.2 and a more recent version of LibreSSL. 2.7 and 3.6 branches still target platforms with ancient versions of OpenSSL (e.g. Ubuntu 14.04 has 1.0.1f + patches). People were complain A LOT, because there were not able to install Python 3.7 on TravisCI. Like really a lot, alot.

I propose to close this bug as fixed in 3.7+

[ned-deily]

New changeset 3dbc43f by Ned Deily (Victor Stinner) in branch '3.6':
bpo-32947: test_ssl fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-11612)
3dbc43f

[ned-deily]

I don't have a strong opinion about backporting to 3.6. With OpenSSL 1.0.2 official support ending at the end of 2019 and 3.6.z retired towards the ned of 2021, there would be a 2-year window where 3.6 is still in security-fix-only status. But, if we don't do the backport now, we could always choose to backport it later if the need arises.