unittest load_tests protocol not working as documented

[stefanseefeld]

BPO	25520
Nosy	@stefanseefeld, @rbtcollins, @ezio-melotti, @voidspace
Files	unittest_bug.tgz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-10-30.21:15:02.003>
labels = ['type-bug', 'library']
title = 'unittest load_tests protocol not working as documented'
updated_at = <Date 2016-03-14.02:37:06.323>
user = 'https://github.com/stefanseefeld'

bugs.python.org fields:

activity = <Date 2016-03-14.02:37:06.323>
actor = 'rbcollins'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-10-30.21:15:02.003>
creator = 'stefan'
dependencies = []
files = ['40905']
hgrepos = []
issue_num = 25520
keywords = []
message_count = 5.0
messages = ['253757', '258716', '258747', '258766', '261716']
nosy_count = 4.0
nosy_names = ['stefan', 'rbcollins', 'ezio.melotti', 'michael.foord']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25520'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[stefanseefeld]

As described in the README contained in the attached tarball, I'm observing wrong behavior. I have based this code on my understanding of https://docs.python.org/2/library/unittest.html#load-tests-protocol, but the effect isn't as expected (I see duplicate appearances of tests whenever I use the load_tests() mechanism.)

[stefanseefeld]

Hi, I'm investigating more issues related to test loading, and thus I have discovered issue bpo-16662.

I have found quite a number of inconsistencies and bugs in the loading of tests. But without getting a sense of what the supposed behavior is I find it difficult to come up with workarounds and fixes.

Is there a place where these issues can be discussed (rather than just looking at each bug individually) ?

I'd ultimately like to be able to invoke

python -m unittest my.test.subpackage

and have unittest pick up all the tests within that recursively, with and without load_tests() functions.

(On a tangential note, I would also like to have a mode where the found tests are listed without being executed. I have hacked a pseudo-TestRunner that does that, but I'm not sure this is the best approach.)

Is there any place where the bigger picture can be discussed ?

Thanks,

[voidspace]

To find and run tests recursively from a test package you want test discovery rather than (necessarily) load_tests.

For a bug report, please describe the behaviour you expect and the behaviour you're seeing instead.

[stefanseefeld]

I believe what I actually want is for the discovery mechanism to be fully implicit. It turns out that already *almost* works right now.

What doesn't work (and what this bug report really was about initially), is the use of the 'discover' command with the '-p "*.py"' argument, which for some reason makes certain tests (all ?) count twice. It looks like packages are visited twice, once as modules, and once via their contained '__init__.py' file...

(For the implicit discovery to work better, I believe, the discovery-specific options need to be made available through the main parser, so they can be used even without the 'discover' command.)

[rbtcollins]

What version of python are you testing with? unittest 2.7 has a number of bugs vis-a-vis namespaces and discovery. If you're testing with less than 3.5, or perhaps 3.6, please try with unittest2, which has the same fixes as the stdlib unittest.