regrtest checks (os.environ, sys.path, etc.) are hard to use

[merwok]

BPO	12314
Nosy	@vstinner, @ezio-melotti, @merwok, @bitdancer, @voidspace

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-10-02.21:24:02.741>
created_at = <Date 2011-06-10.16:19:09.319>
labels = ['type-feature', 'tests']
title = 'regrtest checks (os.environ, sys.path, etc.) are hard to use'
updated_at = <Date 2015-10-02.21:24:02.740>
user = 'https://github.com/merwok'

bugs.python.org fields:

activity = <Date 2015-10-02.21:24:02.740>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-10-02.21:24:02.741>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2011-06-10.16:19:09.319>
creator = 'eric.araujo'
dependencies = []
files = []
hgrepos = []
issue_num = 12314
keywords = []
message_count = 9.0
messages = ['138101', '138109', '138110', '138378', '138381', '138673', '144276', '145110', '220508']
nosy_count = 7.0
nosy_names = ['vstinner', 'ezio.melotti', 'eric.araujo', 'r.david.murray', 'michael.foord', 'BreamoreBoy', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12314'
versions = ['Python 3.3']

[merwok]

When regrtest tells me that os.environ was modified by test_packaging, it’s very hard to find which test is responsible. The -vv output is hard to read: it prints four dictionaries, which I have to visually delineate, copy, paste, pretty-print and diff to find what’s changed. Then I edit the source code of regrtest so that it fails earlier, or I add a debug print in the packaging.util code that touches os.environ.

I’d like regrtest to tell me what exactly was changed, and where. For the first request, I think each checked object could use a type-specific method, for example _compare_dict would see if any value was changed and if keys are different, and print only that info.

[bitdancer]

Improvements are welcome, the whole thing was a QnD hack that has been evolving over time...maybe there's some way to reuse unittest's assertEqual machinery?

I also wonder if the whole environment-changed thing should be moved into unittest so it could wrap around individual test methods.

[merwok]

maybe there's some way to reuse unittest's assertEqual machinery?
Hum, I think the diff functionality is cleanly separated in a (private) method, so we could get diff for free.

I also wonder if the whole environment-changed thing should be moved
into unittest so it could wrap around individual test methods.
Yeah, I was thinking about a WatchfulTestRunner that would perform the same checks. It annoys me to have two diverging sets of features to run tests: either checks with regrtest, or failfast, discovery and granularity with unittest.

[voidspace]

I don't think this is something that belongs in unittest - it's not something particularly useful (or at least particularly requested) outside of the python test suite. No reason that a WatchfulTestRunner couldn't live in regrtest.

[bitdancer]

Sounds like a plan.

[vstinner]

I’d like regrtest to tell me what exactly was changed, and where.

regrtests has many tests (you give some examples: os.environ, sys.path), run all tests after calling a single test function would make regrtest slower. We can add an option (e.g. --strict?) to run these sanity tests after calling a single test function.

I would also be nice to have this option for the refleak checks: run refleak checks on each function, instead of doing the checks on the whole file.

[python-dev]

New changeset 9c61f46ea6b4 by Éric Araujo in branch 'default':
Make regrtest look at internal dicts in sysconfig.
http://hg.python.org/cpython/rev/9c61f46ea6b4

[merwok]

Idea: when a regrtest check fails, the test is run again but a WatchfulMixin is injected into the test case’s bases, so that setUp and tearDown call the right methods. This would be more efficient than always running with a watchful test runner.

[BreamoreBoy]

@Éric do you intend following up on this?