regrtest: re-run failed tests in a subprocess

[vstinner]

BPO	33680
Nosy	@vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-19.23:56:46.194>
created_at = <Date 2018-05-29.11:41:03.086>
labels = ['3.8', 'tests']
title = 'regrtest: re-run failed tests in a subprocess'
updated_at = <Date 2018-09-19.23:56:46.193>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-09-19.23:56:46.193>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-19.23:56:46.194>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2018-05-29.11:41:03.086>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 33680
keywords = []
message_count = 2.0
messages = ['318011', '318013']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33680'
versions = ['Python 3.8']

[vstinner]

The -w option of regrtest (python -m test) re-run failed tests in verbose in the current process. Problem: if the test does crash, regrtest doesn't re-run following tests, it doesn't write the final tests result (like failing tests, total duration, etc.).

To make regrtest tolerant to such crashes, re-run tests should be run in verbose mode. The only constraint is to disable buffering on stdout and stderr to make sure that stdout and stderr order is kept in pipes. Or maybe stderr should be redirected to stdout to avoid ordering issue?

[vstinner]

A random example of such crash, process killed by faulthandler timeout:
http://buildbot.python.org/all/#/builders/118/builds/65

(...)
394 tests OK.

10 slowest tests:

• test_subprocess: 7 min 13 sec
• test_concurrent_futures: 2 min 47 sec
• test_multiprocessing_spawn: 2 min 29 sec
• test_urllib2net: 1 min 45 sec
• test_multiprocessing_forkserver: 1 min 34 sec
• test_nntplib: 1 min 29 sec
• test_smtpnet: 1 min 23 sec
• test_multiprocessing_fork: 1 min 22 sec
• test_tools: 1 min 19 sec
• test_tokenize: 1 min 13 sec

1 test failed:
test_asyncio

20 tests skipped:
test_dbm_gnu test_devpoll test_epoll test_gdb test_idle test_ioctl
test_msilib test_ossaudiodev test_spwd test_startfile test_tcl
test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle
test_winconsoleio test_winreg test_winsound test_zipfile64
Re-running failed tests in verbose mode
Re-running test 'test_asyncio' in verbose mode
(...)
test_sock_client_ops (test.test_asyncio.test_events.PollEventLoopTests) ... ok

Timeout (0:15:00)!
Thread 0x000000080139a000 (most recent call first):
File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 1713 in _run_once
File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 523 in run_forever
File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 555 in run_until_complete
File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/test_asyncio/test_events.py", line 2135 in run_loop
File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/test_asyncio/test_events.py", line 2249 in test_sock_sendfile_mix_with_regular_send
(...)

test_sock_sendfile_mix_with_regular_send (test.test_asyncio.test_events.PollEventLoopTests) ... *** Error code 1
Stop.
make: stopped in /usr/home/buildbot/python/3.7.koobs-freebsd-current/build
program finished with exit code 1
elapsedTime=2612.384935