[2.7] regrtest announces 401 tests, but there are 304 tests?

[vstinner]

BPO	30254
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @serhiy-storchaka, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-03.14:31:53.714>
created_at = <Date 2017-05-03.10:44:11.494>
labels = ['tests', 'OS-windows']
title = '[2.7] regrtest announces 401 tests, but there are 304 tests?'
updated_at = <Date 2017-05-03.14:31:53.713>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-05-03.14:31:53.713>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-03.14:31:53.714>
closer = 'vstinner'
components = ['Tests', 'Windows']
creation = <Date 2017-05-03.10:44:11.494>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30254
keywords = ['buildbot']
message_count = 4.0
messages = ['292886', '292887', '292895', '292905']
nosy_count = 6.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30254'
versions = ['Python 2.7']

[vstinner]

First case:
---
...
[301/401] test_with
[302/401] test_import
[303/401] test_xmlrpc
277 tests OK.
2 tests altered the execution environment:
test_distutils test_sax
24 tests skipped:
test_al test_bsddb185 test_cl test_commands test_curses test_dbm
test_dl test_fork1 test_grp test_ioctl test_kqueue test_macos
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pwd test_readline test_scriptpackages test_spwd
test_threadsignals test_wait3 test_wait4
1 skip unexpected on win32:
test_readline
---
http://buildbot.python.org/all/builders/AMD64%20Windows10%202.7/builds/133/steps/test/logs/stdio

Why does regrtest count 404 tests, whereas the total is 304 tests (277+2+24+1)?

Second case:
---
...

[212/401] test_extcall
[213/401] test_frozen
[214/401] test_hotshot
190 tests OK.
2 tests altered the execution environment:
test_distutils test_sax
22 tests skipped:
test_al test_applesingle test_bsddb185 test_crypt test_curses
test_dbm test_dl test_fcntl test_fork1 test_gdb test_gl test_mhlib
test_nis test_openpty test_ossaudiodev test_pipes test_poll
test_pwd test_resource test_scriptpackages test_threadsignals
test_wait4
1 skip unexpected on win32:
test_gdb
---
http://buildbot.python.org/all/builders/AMD64%20Windows8%202.7/builds/128/steps/test/logs/stdio

Here I only count 215 tests (190+2+22+1)... why not 304 or 401? where are the remaining tests?

On Linux, it seems fine:
---
...
[399/401] test_sets
[400/401] test_bufio
[401/401] test_xpickle
371 tests OK.
30 tests skipped:
test_aepack test_al test_applesingle test_bsddb test_bsddb185
test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_gdb
test_gl test_imgfile test_kqueue test_macos test_macostools
test_msilib test_py3kwarn test_scriptpackages test_startfile
test_sunaudiodev test_tk test_tools test_ttk_guionly test_winreg
test_winsound test_zipfile64
9 skips unexpected on linux2:
test_bsddb test_bsddb3 test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_gdb
test_tools
---
http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%202.7/builds/129/steps/test/logs/stdio

I also expects regrtest to ends with "[401/401]" on Windows.

[vstinner]

Another example, worse:
---
...
[123/401] test_source_encoding
[124/401] test_locale
[125/401] test_unary
116 tests OK.
9 tests skipped:
test_applesingle test_bsddb185 test_ioctl test_kqueue
test_linuxaudiodev test_msilib test_startfile test_sunaudiodev
test_tk
1 skip unexpected on linux2:
test_ioctl
---
http://buildbot.python.org/all/builders/PPC64LE%20Fedora%202.7/builds/127/steps/test/logs/stdio

Only 116 tests here???

[vstinner]

Maybe be related to issue bpo-30255 (test_xml_etree crash related to slices).

[vstinner]

It has been long time since I didn't work on Python 2.7, so I forgot that regrtest of Python 2.7 lacks *many* features of Python master. I created the issue bpo-30258 to add code to handle crash of a test child process.

Maybe be related to issue bpo-30255 (test_xml_etree crash related to slices).

regrtest stopped at the first crash of a child process and swallow child stdout/stderr. This crash explains this issue. Since issue bpo-30255 is fixed, I close this issue to now focus on the issue bpo-30258 (fix regrtest to detect crashes).