Debian Sid/Buster: Cannot enable TLS 1.0/1.1 with PROTOCOL_TLS

[AdrianVollmer]

BPO	31453
Nosy	@doko42, @tiran, @ned-deily, @njsmith, @AdrianVollmer, @miss-islington, @smartmadre
PRs	bpo-31453: [WIP] Allow to change TLS protocols on Debian #3662 bpo-31453: Add setter for min/max protocol version #5259 [3.7] bpo-31453: Add setter for min/max protocol version (GH-5259) #5926
Files	poc.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-17.19:33:08.562>
created_at = <Date 2017-09-13.15:48:25.120>
labels = ['expert-SSL', 'type-bug']
title = 'Debian Sid/Buster: Cannot enable TLS 1.0/1.1 with PROTOCOL_TLS'
updated_at = <Date 2021-04-17.19:33:08.561>
user = 'https://github.com/AdrianVollmer'

bugs.python.org fields:

activity = <Date 2021-04-17.19:33:08.561>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-17.19:33:08.562>
closer = 'christian.heimes'
components = ['SSL']
creation = <Date 2017-09-13.15:48:25.120>
creator = 'adrianv'
dependencies = []
files = ['47139']
hgrepos = []
issue_num = 31453
keywords = ['patch']
message_count = 30.0
messages = ['302081', '302082', '302084', '302085', '302086', '302088', '302091', '302092', '302093', '302094', '302097', '302099', '302100', '302561', '307105', '310996', '311008', '312920', '312991', '312997', '312998', '314635', '316017', '317534', '317574', '317610', '317618', '317619', '329218', '391309']
nosy_count = 9.0
nosy_names = ['doko', 'christian.heimes', 'ned.deily', 'njs', 'kroeckx', 'Alex Gaynor', 'adrianv', 'miss-islington', 'mabrafoo']
pr_nums = ['3662', '5259', '5926']
priority = 'critical'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31453'
versions = ['Python 2.7', 'Python 3.6']

[AdrianVollmer]

According to the documentation (https://docs.python.org/2/library/ssl.html#ssl.PROTOCOL_TLS), using ssl_version = ssl.PROTOCOL_TLS in a server socket should offer all TLS/SSL versions. However, it only offers TLSv1_2.

I attached a proof of concept.

$ python3 poc.py
3.5.4 (default, Aug 12 2017, 14:08:14)
[GCC 7.1.0]
OpenSSL 1.1.0f  25 May 2017
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:719)
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:719)
b'test\n'

$ python2 poc.py
2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118]
OpenSSL 1.1.0f  25 May 2017
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:661)
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:661)
test

To connect with s_client:

 $ for i in {tls1,tls1_1,tls1_2} ; do echo test | openssl s_client -connect localhost:3333 -CAfile server.pem -quiet -$i ; done
140164347663616:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:../ssl/record/rec_layer_s3.c:1399:SSL alert number 70
139926441944320:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:../ssl/record/rec_layer_s3.c:1399:SSL alert number 70
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
read:errno=0

[AlexGaynor]

What operating system are you on?

[AdrianVollmer]

Debian buster/sid

[tiran]

Debian Buster has patched OpenSSL to disable TLS 1.0 and 1.1 by default,
https://lists.debian.org/debian-devel-announce/2017/08/msg00004.html

[AdrianVollmer]

I read about that, but I don't understand. If I use openssl s_server -port 3333 , I can connect using either one of the three protocols.

Even if that's the new default, is there no way now to get python on Buster/Sid to use OpenSSL in a non-default mode and have it offer all three versions?

[tiran]

You have to enable the protocols by applying a reverse bitmask to SSLContext.options:

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS)
ctx.load_cert_chain('server.pem')
ctx.options &= ~(ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1)

sslsock = ctx.wrap_socket(s, server_side=True)

[AdrianVollmer]

Doesn't seem to do anything:

>>> ctx.options
2181170175L
>>> ctx.options & ~(ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1)
2181170175L

[tiran]

Please report this issue to the Debian maintainers. I don't know how Debian has disabled TLS 1.0 and TLS 1.1 for the SSL_METHOD *TLS_method(void). It might not be possible to enable auto-negotiation for old protocols at all.

[AdrianVollmer]

Okay, thanks for your time!

[tiran]

Ah, here we go: https://anonscm.debian.org/viewvc/pkg-openssl/openssl/branches/1.1.0/debian/patches/tls1_2_default.patch

Debian patched the default for SSL_CTX_set_min_proto_version(). The SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version() API calls are OpenSSL 1.1.0-only and not available from Python. It is not possible to override the minimum version from Python.

[AdrianVollmer]

I have a workaround for now:

        versions = [ssl.PROTOCOL_TLSv1,
                    ssl.PROTOCOL_TLSv1_1,
                    ssl.PROTOCOL_TLSv1_2,
                   ]
        firstbytes = s.recv(16, socket.MSG_PEEK)
        ss = ssl.wrap_socket(
            s,
            server_side=True,
            certfile="server.pem",
            keyfile="server.pem",
            #  ssl_version=versions[ord(firstbytes[10])-1] # python2
            ssl_version=versions[firstbytes[10]-1]
        )

How much of an ugly hack is this? :)

[tiran]

It's an ugly hack and not a long term solution. The PROTOCOL_TLSv* constants and ssl.wrap_socket() are discouraged and will be removed soon.

[tiran]

Matthias, this issue affects Debian and probably Ubuntu, too. Could you please discuss it with Debian maintainers and propose a workaround? Python does not expose the new OpenSSL 1.1.0 SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version() calls. We only support SSL_CTX_set_options() with SSL_OP_NO_TLSv1 and SSL_OP_NO_TLSv1_1.

[tiran]

PR 3662 undos Debian's patching of OpenSSL. I'm not keen to undo a security improvement. However Debian is breaking backwards compatibility. For Python 3.7 we could consider to disable TLS 1.0 and TLS 1.1 for PROTOCOL_TLS_SERVER and PROTOCOL_TLS_CLIENT.

[ned-deily]

FWIW, Debian seems to have re-enabled TLS 1.0 and 1.1 in "testing". As a result, test_ssl now passes again.

openssl (1.1.0g-1) unstable; urgency=medium

• New upstream version
  • Fixes CVE-2017-3735
  • Fixes CVE-2017-3736
• Remove patches applied upstream
• Temporary enable TLS 1.0 and 1.1 again (bpo-875423)
• Attempt to fix testsuite race condition
• update no-symbolic.patch to apply

-- Kurt Roeckx <kurt@roeckx.be> Thu, 02 Nov 2017 15:22:48 +0100

[ned-deily]

I'm not clear if this is still needed, i.e. has Debian backed off on their change across the board? If it is still needed, I'm going to allow an extension for landing of it until 3.7.0b2, currently scheduled for 2018-02-26. If anyone else can help Christian get this in before b2, that would be great. I'm removing older versions for now. We can discuss potential backports after the code lands.

[tiran]

The feature is still useful -- whether or not Debian disables TLS 1.0 and 1.1. The new API is easier to use and more convenient than the old bitmask approach. "option &=~OP_NO_SSLv3" is just horrible. :)

[tiran]

I have closed the feature newer BPO-32609 in favor of this bug because Ned gave this bug a deferred blocker priority.

OpenSSL 1.1 has introduced a new API to set the minimum and maximum supported protocol version. The API is easier to use than the old OP_NO_TLSv1 option flags, too

https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_min_proto_version.html

Debian used the new setters to disable TLS 1.0 and 1.1 in testing, bpo-31453. The old TLS versions have been enabled again for now. Python must expose the new API in case Debian decides to disable them again. Another $DIST has considered to implement a virtually the same policy as Debian.

I also like to deprecate the old OP_NO_TLSv1 et al. flags in favor of the new API. The option flags are awkward to use and easy to get wrong. For example applications must not leave holes in the OP_NO range (e.g. allow TLS 1.0 and 1.2 but disable 1.1).

[tiran]

I have another good reason to land PR 5259 in 3.7. OpenSSL 1.1.0 has deprecated the old way to disable/enable protocol versions with set option. The OP_NO_TLSv1* constants will likely get removed in OpenSSL 1.2.0. I'm expecting to see a 1.2.0 release within the next two years. Therefore we should include SSLContext.maximum_version and SSLContext.minimum_version now. We may even have to backport them to 3.6 and 2.7.

https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_options.html

SSL_OP_NO_SSLv3, SSL_OP_NO_TLSv1, SSL_OP_NO_TLSv1_1, SSL_OP_NO_TLSv1_2, SSL_OP_NO_DTLSv1, SSL_OP_NO_DTLSv1_2
These options turn off the SSLv3, TLSv1, TLSv1.1 or TLSv1.2 protocol versions with TLS or the DTLSv1, DTLSv1.2 versions with DTLS, respectively. As of OpenSSL 1.1.0, these options are deprecated, use SSL_CTX_set_min_proto_version and SSL_CTX_set_max_proto_version instead.

[miss-islington]

New changeset 4c842b0 by Miss Islington (bot) in branch '3.7':
bpo-31453: Add setter for min/max protocol version (GH-5259)
4c842b0

[tiran]

Ned,

I have pushed the enhancement now. I feel a bit guilty about the last minute push, but I honestly believe it's in our best interesting.

There is a high chance that the new APIs will be required for Debian and other distros in the near future. OpenSSL has deprecated the old API, too. It is likely that OpenSSL 1.2.0 will remove both the options (OP_NO_TLSv1, OP_NO_TLSv1_2, ...) options and version specific protocols (PROTOCOL_TLSv1, PROTOCOL_TLSv1_2, ...).

[ned-deily]

Christian, is there more needed for this issue or can it be closed?

[ned-deily]

Christian, ping. Can we close this?

[ned-deily]

Christian, ping again: can this be closed? In any case, it would not seem to be a "deferred blocker"; downgrading to "critical". Please close or update, thanks!

[tiran]

The problem no longer affects Python 3.7 and 3.8. It may affects Python 3.6 and 2.7 if Debian to decide to disable TLS 1.0 and 1.1 again. If Debian uses the new OpenSSL 1.1.0 API to disable the protocols, then I have to backport https://bugs.python.org/issue32609 to 2.7 and 3.6.

[kroeckx]

Note that the version in experimental only supports TLS 1.2 and 1.3 with the default config. It's moved from fixed in the code, to the default config file. I expect to upload that to unstable "soon", at which point people will be affected by this again.

[tiran]

Thanks Kurt,

which API are you using to disable TLS 1.0 and 1.1? Is it the old SSL_CTX_set_options() or the new SSL_CTX_set_min/max_proto_version() API?

[kroeckx]

The effect is the same as calling SSL_CTX_set_min_proto_version().

[smartmadre]

FYI, This is how I figured out and fixed the issue on my debian system.
-Run nmap to figure out what ssl version is being used by the server
nmap -p443 -sV --script ssl-enum-ciphers 10.10.10.7
output says TLSv1.0

test 10.10.10.7 using example in this comment (gets expected error)
psf/requests#606 (comment)

test with openssl binary (gets expected error)
openssl s_client -connect 10.10.10.7:443

fix by editing this value-->MinProtocol = TLSv1.0
in this file--> /etc/ssl/openssl.cnf

rerun tests without error.
Note the outdated server I am connecting to is internal, non-production, not connected to the internet.

[tiran]

Python 2.7 is out of supports. 3.6 will reach end of security support soon. More recent Python versions have TLS 1.0 and 1.1 deprecated and contain workarounds for tests.