test_asyncio fails on RHEL8, or on Fedora using NEXT security policy

[stratakis]

BPO	35352
Nosy	@vstinner, @asvetlov, @1st1, @stratakis
PRs	bpo-35352 Use the certificate set from the test directory for test_asyncio #10826 [3.7] bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) #10830 bpo-35352: Cleanup test_asyncio/utils.py #10831 [3.6] bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) #10832 [3.7] bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) #10834
Files	asyncio_failures_3.8.log asyncio_failures_3.7.log asyncio_failures_3.6.log

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-30.22:00:19.755>
created_at = <Date 2018-11-29.16:13:50.599>
labels = ['3.8', '3.7', 'tests', 'expert-asyncio']
title = 'test_asyncio fails on RHEL8, or on Fedora using NEXT security policy'
updated_at = <Date 2018-11-30.22:00:19.753>
user = 'https://github.com/stratakis'

bugs.python.org fields:

activity = <Date 2018-11-30.22:00:19.753>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-30.22:00:19.755>
closer = 'vstinner'
components = ['Tests', 'asyncio']
creation = <Date 2018-11-29.16:13:50.599>
creator = 'cstratak'
dependencies = []
files = ['47955', '47956', '47957']
hgrepos = []
issue_num = 35352
keywords = ['patch']
message_count = 16.0
messages = ['330698', '330699', '330767', '330787', '330788', '330790', '330803', '330804', '330808', '330812', '330813', '330814', '330818', '330819', '330820', '330828']
nosy_count = 4.0
nosy_names = ['vstinner', 'asvetlov', 'yselivanov', 'cstratak']
pr_nums = ['10826', '10830', '10831', '10832', '10834']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35352'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[stratakis]

When compiling python3 on RHEL8 and run the tests, test_asyncio fails with 8 failures (and more on older branches).

openssl version is 1.1.1

Attaching the failures for the various branches.

[stratakis]

Seems that 3.8 and 3.7 get the same failures, while 3.6 gets 14 instead of 8.

[asvetlov]

The strange thing is these tests are passed on our build bots.

In logs I see only ConnectionResetError. Can it be related to your configuration?

[stratakis]

It seems I can reproduce it on Fedora as well by setting stronger crypto defaults through 'update-crypto-policies --set FUTURE'.

Repo located here: https://gitlab.com/redhat-crypto/fedora-crypto-policies/tree/master

The changes are many, but if I compare with RHEL8, the minimal changes that could affect it are:

-# DH params size: >= 1023
+# DH params size: >= 2048

-# TLS protocols: TLS >= 1.0
+# TLS protocols: TLS >= 1.2, DTLS >= 1.2

-@protocol_list = ('TLS1.3', 'TLS1.2', 'TLS1.1', 'TLS1.0', 'DTLS1.2', 'DTLS1.0');
+@protocol_list = ('TLS1.3', 'TLS1.2', 'DTLS1.2');

- $min_tls_version = 'TLS1.0';
- min_dtls_version = 'DTLS1.0';
+ $min_tls_version = 'TLS1.2';
+ $min_dtls_version = 'DTLS1.2';

# Parameter sizes

• $min_dh_size = 1023;
  + $min_dh_size = 2048;

[asvetlov]

Maybe this is the reason.
asycio test suite uses these ssl contexts: https://github.com/python/cpython/blob/master/Lib/test/test_asyncio/utils.py#L72-L92

Maybe bumping used protocol version will help to pass tests on your box.
Would you try it?

[stratakis]

I believe I figured out the issue, at least on the master branch.

While checking the certificates used by asyncio tests within the test_asyncio folder I noticed they were quite outdated when compared to the more recent updated ones with the test/ folder, which take into account the stronger crypto defaults introduced in the latest openssl versions.

And by looking at 6d8c1ab#diff-a8e7dbb528601706db0f01d01332bb76 it seems that those certs are just copied from test/ within test_asyncio/. So by copying over the old certs, the tests actually pass.

The immediate workaround would be to just copy over the certs but a better approach would be to just reuse the certs within the test/ folder instead of relying on copying them over to test_asyncio/

[vstinner]

The strange thing is these tests are passed on our build bots.

RHEL8 has a very strict security policy by default. I'm not sure if any OS run on buildbot has a security policy as strict as RHEL8?

Maybe bumping used protocol version will help to pass tests on your box.

I tried to tune the SSLContext in many different ways but it doesn't work. The problem comes from the .pem files.

I confirm that copying .pem files from Lib/test/ into Lib/test/test_asyncio/ does fix the issue.

And by looking at 6d8c1ab#diff-a8e7dbb528601706db0f01d01332bb76 it seems that those certs are just copied from test/ within test_asyncio/. So by copying over the old certs, the tests actually pass.

In this case, I don't see the point of having two copies of the same files.

PR 10826 does the right fix: remove .pem files from Lib/test/test_asyncio/ and reuse .pem files from Lib/test/.

[stratakis]

Also on Fedora the same set of security policies can be set as RHEL8 by utilizing 'update-crypto-policies --set NEXT'

[vstinner]

New changeset b062ba7 by Victor Stinner (stratakis) in branch 'master':
bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826)
b062ba7

[asvetlov]

IIRC two copies exist from very early development times when asyncio was not a part of Python stdlib.

[vstinner]

IIRC two copies exist from very early development times when asyncio was not a part of Python stdlib.

Yeah, that was my guess as well. Maybe data_file() could be simplified or replaced by support.findfile(), but I chose the easy solution (minimize changes) :-)

[vstinner]

New changeset 7212148 by Victor Stinner in branch 'master':
bpo-35352: Cleanup test_asyncio/utils.py (GH-10831)
7212148

[vstinner]

New changeset 38bed78 by Victor Stinner in branch '3.7':
[3.7] bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) (GH-10834)
38bed78

[vstinner]

New changeset 02250e5 by Victor Stinner in branch '3.6':
bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) (GH-10832)
02250e5

[vstinner]

I leave the issue open until someone validates manually that the issue os fixed in all branches with stricter security (ex: RHEL8).

[vstinner]

I tested on Fedora 29 using:

sudo update-crypto-policies --set NEXT

With this config, I was able to reproduce the test_asyncio failure on 3.6, 3.7 and master branches.

I confirm that the commits fixed test_asyncio in these 3 branches. Thanks Charalampos Stratakis!