Add regrtest option to record test results to a file

[brettcannon]

BPO	13954
Nosy	@brettcannon

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-10-02.21:23:34.451>
created_at = <Date 2012-02-06.20:35:12.682>
labels = ['type-feature', 'tests']
title = 'Add regrtest option to record test results to a file'
updated_at = <Date 2015-10-02.21:23:34.450>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2015-10-02.21:23:34.450>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-10-02.21:23:34.451>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2012-02-06.20:35:12.682>
creator = 'brett.cannon'
dependencies = []
files = []
hgrepos = []
issue_num = 13954
keywords = []
message_count = 2.0
messages = ['152771', '221809']
nosy_count = 3.0
nosy_names = ['brett.cannon', 'BreamoreBoy', 'tshepang']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13954'
versions = ['Python 3.5']

[brettcannon]

The idea is that if a test succeeded then it is written to a file commented out(including skipped tests w/ the appropriate comment), and if a test failed then it is left uncommented. This way the failing tests can simply be passed to --fromfile for easy repeatability. This becomes especially handy when randomizing test order and there is some dependency on a previously run test as you can begin to uncomment tests until you find the trigger.

This can also be used with some other new flag which lists which tests to skip (i.e. the inverse of --fromfile). This is helpful when you don't want to run tests in some branch when you know they are failing in the default branch already and you are not trying to fix them.

[BreamoreBoy]

@brett I assume that you'd want to follow up on this.