Add decorator for tests not yet implemented

[zware]

BPO	19597
Nosy	@pitrou, @ezio-melotti, @voidspace, @zware, @serhiy-storchaka
Files	support.not_implemented.diff support.not_implemented.v2.diff: Add regrtest option

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zware'
closed_at = <Date 2013-11-19.04:02:38.636>
created_at = <Date 2013-11-14.22:33:12.266>
labels = ['type-feature', 'tests']
title = 'Add decorator for tests not yet implemented'
updated_at = <Date 2013-11-19.04:02:38.635>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2013-11-19.04:02:38.635>
actor = 'zach.ware'
assignee = 'zach.ware'
closed = True
closed_date = <Date 2013-11-19.04:02:38.636>
closer = 'zach.ware'
components = ['Tests']
creation = <Date 2013-11-14.22:33:12.266>
creator = 'zach.ware'
dependencies = []
files = ['32626', '32629']
hgrepos = []
issue_num = 19597
keywords = ['patch']
message_count = 5.0
messages = ['202907', '202919', '203089', '203093', '203337']
nosy_count = 5.0
nosy_names = ['pitrou', 'ezio.melotti', 'michael.foord', 'zach.ware', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19597'
versions = ['Python 3.3', 'Python 3.4']

[zware]

Some tests in the test suite are not implemented for one reason or another, and most of these are defined simply as "def test_thats_not_implemented(self): pass", possibly with a comment meant to be a reminder to implement it. This patch adds a decorator to test.support which turns the non-test into an expected failure with a docstring. This means that when run in verbose mode, instead of showing:

"""
test_thats_not_implemented (test.test_sometest.TestClass) ... ok
"""

it will instead show:

"""
Not Implemented: TestClass.test_thats_not_implemented ... expected failure
"""

This should make it more obvious that such a test needs some work. The patch also applies the decorator in test_minidom as an example; there are a few other places that could use it as well.

[zware]

This patch takes the idea a little further, adding a command line option to regrtest, '--failnoimpl' (it could use a better name).

With this option specified any test decorated by support.not_implemented will not be marked as an expected failure, allowing the NotImplementedError exception to fail the test. The default under regrtest is to expect the failure, but the default for running a test file directly is to fail loudly. The idea is just to make it more obvious when a test is just in need of implementation, and could make it easier for new contributors to find something that needs doing.

[ezio-melotti]

I don't think a new option for regrtest is necessary.
I'm also not sure expected failure is a good idea -- in some cases it might be better to report a skip or an error.
Do you have other examples where this can be used? If there aren't many occurrences we might also fix them and avoid adding a new decorator.

[pitrou]

Indeed, this doesn't sound like a terrific idea to me.

[zware]

There are a couple in pickletester, one in test_io, a few in test_reprlib, and the ones in the patch from test_minidom; some 10 test methods in total that I've found and remembered. I suppose with that few, it would be best to just open issues for each test module and have them implemented. Most of them will be getting explicit skips as part of bpo-19572 anyway.

This would tend to make it easier to get away with checking in new unimplemented tests, which isn't really a good thing. Closing as rejected.