regrtest: add -k and -K options to filter tests by function/file names

[vstinner]

BPO	12231
Nosy	@pitrou, @vstinner, @ezio-melotti, @merwok, @bitdancer
Superseder	bpo-12626: run test cases based on a glob filter
Files	regrtest_regex.patch regrtest_regex-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-12-18.16:42:09.925>
created_at = <Date 2011-05-31.23:56:12.967>
labels = ['type-feature', 'tests']
title = 'regrtest: add -k and -K options to filter tests by function/file names'
updated_at = <Date 2011-12-18.16:42:09.923>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2011-12-18.16:42:09.923>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2011-12-18.16:42:09.925>
closer = 'pitrou'
components = ['Tests']
creation = <Date 2011-05-31.23:56:12.967>
creator = 'vstinner'
dependencies = []
files = ['22216', '22538']
hgrepos = []
issue_num = 12231
keywords = ['patch']
message_count = 11.0
messages = ['137402', '137405', '137429', '137433', '137525', '137589', '137640', '139581', '141032', '141130', '149781']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'ezio.melotti', 'eric.araujo', 'Arfrever', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '12626'
type = 'enhancement'
url = 'https://bugs.python.org/issue12231'
versions = ['Python 3.3']

[vstinner]

Attached patch adds -k/--func-regex and -K/--file-regex options to filter tests by function/file names. It modifies makeSuite() and getTestCaseNames() functions of unittest.loader to add an optional filter argument: callback taking a function name as argument and returning False if the test should be ignored. You can use the new options more than once and their argument are regular expressions (case insensitive).

Examples:

• "./python -m test -k subprocess -K pass_fds" runs only 1 function of 1 file: test_subprocess.test_pass_fds()
• "./python -m test -k os$" runs 1 file: test_os
• "./python -m test -K codecencoding" runs 6 files: test_codecencodings_cn, test_codecencodings_hk, test_codecencodings_iso2022, test_codecencodings_jp, test_codecencodings_kr, test_codecencodings_tw

[vstinner]

Attached patch adds -k/--func-regex and -K/--file-regex options to filter
tests by function/file names.

We need maybe a third option to filter tests by their class name. And we need
also maybe the opposite option: exclude tests matching a regex. Another
proposition: --include and --exclude options using the following format:

<file regex>:<class regex>:<function regex>

Each field is optional, examples:

--include "test_subprocess::leak" ignores test_leak function of
test_subprocess file
--include "codecencoding::" runs the 6 test_codecencoding* files
--include "test_subprocess:^C:" runs only the tests of the io module (not the
_pyio module)

• etc.

Problem: --exclude (-x) option is already used.

[ezio-melotti]

IMHO it would be better to tweak regrtest so that it understand test_file.TestClass.test_method, possibly supporting wildcards.
E.g.:

• ./python -m test test_subprocess.POSIXProcessTestCase
• ./python -m test test_subprocess.POSIXProcessTestCase.test_pass_fds
• ./python -m test '*codecencoding*'
• ./python -m test 'test_subprocess.C*'

The -x/--exclude option could be extended to support this notation too and make possible to exclude specific tests classes/methods.
Wildcards are less flexible than regex, but I think they are good enough for our use case and simpler to use.

[bitdancer]

+1 to what Ezio said, though I'd rather have this than not have the functionality at all :)

[merwok]

How about reusing unittest discovery in regrtest?

[vstinner]

How about reusing unittest discovery in regrtest?

Does this feature support filtering by keyword for file names and function names?

[merwok]

File name, class name, method name unless I misremember.

[vstinner]

regrtest_regex-2.patch: minor update, just ensure that the patch applies correctly on the default branch.

[ezio-melotti]

See also bpo-12626 for a similar suggestion.

[pitrou]

This patch doesn't work if run_unittest() is called with an already constructed test suite or test case (the patch in bpo-12626 does).

[pitrou]

Already fixed in bpo-12626.