aarch64 RHEL7 LTO + PGO 3.7: "make" hangs when running test_asyncio

[vstinner]

BPO	43031
Nosy	@vstinner
PRs	bpo-43031: Set a timeout when running tests in PGO build #24339

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-27.10:20:17.723>
created_at = <Date 2021-01-26.13:46:19.856>
labels = ['3.7', 'tests']
title = 'aarch64 RHEL7 LTO + PGO 3.7: "make" hangs when running test_asyncio'
updated_at = <Date 2021-01-27.10:20:17.723>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-01-27.10:20:17.723>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-27.10:20:17.723>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2021-01-26.13:46:19.856>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43031
keywords = ['patch']
message_count = 5.0
messages = ['385712', '385716', '385724', '385752', '385753']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['24339']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43031'
versions = ['Python 3.7']

[vstinner]

test_asyncio hangs randomly on Python 3.7 on aarch64 RHEL7 LTO + PGO 3.7. The symptom is a failed build failing with:

retry lost connection compile (retry)

Full error:

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.

Example of failed build:

https://buildbot.python.org/all/#/builders/42/builds/4703

Configure command:

./configure --prefix '$(PWD)/target' --with-lto --enable-optimizations

Compile command:

make -j10 all

End of the make output:
--------
make run_profile_task
make[1]: Entering directory `/home/buildbot/buildarea/3.7.cstratak-RHEL7-aarch64.lto-pgo/build'
./python -m test.regrtest --pgo || true
0:00:00 load avg: 1.07 Run tests sequentially
0:00:00 load avg: 1.07 [ 1/416] test_grammar
(...)
0:01:40 load avg: 1.07 [ 24/416] test_asynchat
0:01:42 load avg: 1.07 [ 25/416] test_asyncio

remoteFailed: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
]
--------

I ran manually "./configure --prefix '$(PWD)/target' --with-lto --enable-optimizations && make -j10 all" twice on the worker directly, but test_asyncio passed successfully (I tried it twice).

[vstinner]

Python 3.7 no longer accept bugfixes, only security fixes:
https://devguide.python.org/#status-of-python-branches

Maybe we should just remove this PGO worker (but keep other Python 3.7 workers).

[vstinner]

I created python/buildmaster-config#228 to skip PGO builds on Python 3.7, since test_asyncio hangs on PGO builds.

[vstinner]

New changeset 6790005 by Victor Stinner in branch 'master':
bpo-43031: Set a timeout when running tests in PGO build (GH-24339)
6790005

[vstinner]

I close the issue.

I added a timeout (20 min) in master when running tests for a PGO build.

I created python/buildmaster-config#228 to skip PGO builds on Python 3.7, since test_asyncio hangs on PGO builds.

Done. I remove all Python 3.7 PGO buildbot builders.

The test_asyncio issue on Python 3.7 looks like a race condition which is likely already fixed in Python 3.8. Even if it's a bug, we cannot fix asyncio bugs in Python 3.7 anymore.