Test discovery for unittest

[voidspace]

BPO	6001
Nosy	@gpshead, @pitrou, @benjaminp, @voidspace
Files	test_discovery.patch test_discovery.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/voidspace'
closed_at = <Date 2009-05-29.20:44:19.885>
created_at = <Date 2009-05-11.21:40:35.814>
labels = ['type-feature', 'library']
title = 'Test discovery for unittest'
updated_at = <Date 2009-05-29.20:44:19.883>
user = 'https://github.com/voidspace'

bugs.python.org fields:

activity = <Date 2009-05-29.20:44:19.883>
actor = 'michael.foord'
assignee = 'michael.foord'
closed = True
closed_date = <Date 2009-05-29.20:44:19.885>
closer = 'michael.foord'
components = ['Library (Lib)']
creation = <Date 2009-05-11.21:40:35.814>
creator = 'michael.foord'
dependencies = []
files = ['13961', '14052']
hgrepos = []
issue_num = 6001
keywords = ['patch', 'needs review']
message_count = 7.0
messages = ['87591', '87616', '88268', '88271', '88434', '88448', '88536']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'pitrou', 'benjamin.peterson', 'michael.foord']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6001'
versions = ['Python 2.7', 'Python 3.2']

[voidspace]

Attached is a patch that implements test discovery for unittest.

It includes command line argument handling (awkward manual handling but
works fine...), so that it can be invoked through:

python -m unittest discover
python -m unittest discover start_dir pattern top_level_dir

If the patch is acceptable then I will write docs. Do I need to make a
post to Python-dev or is it ok to have it reviewed here? The basic
mechanism was hammered out on the Testing-in-Python mailing list with a
lot of input from Robert Collins.

Brett Cannon has already looked over the test discovery code (not the
command line handling code though!).

It includes a customization hook both for modules and packages to
customize test loading and discovery. Described below:

Advantage of this: it is simple (easy to extend)
Disadvantage: it is simple (a lot it doesn't do)

Test discovery is provided through the TestLoader.discover method.

It takes three arguments - start directory, pattern to match test files
(defaults to 'test*.py') and the top level directory of the project
(defaults to the start directory).

The main restriction is that all your tests must be importable from the
top level directory of your project. The start directory is then
recursively searched for files and packages that match the pattern you
pass in. Tests are loaded from matching modules, and all tests run.

(A further restriction is that discovery only currently recognises
packages as directories containing __init__.py files and not .pyc etc.)

The load_tests protocol provides a way for packages and modules to
customize test loading. This is inspired by a similar system in use in
the Bzr test suite.

Iff a test module defines a load_tests function then
TestLoader.loadTestsFromModule will call this function with loader,
tests, None. This should return a suite.

An example 'do nothing' implementation of load_tests would be:

   def load_tests(loader, tests, pattern):
       return tests

If a package directory name matches the pattern you pass into discovery
and the __init__.py contains a load_tests function then it will be
called with loader, tests, pattern. No further discovery will be done
into the package, but because it is passed the pattern as an argument it
is free to continue discovery itself. A do nothing load_tests for a
package is:

    def load_tests(loader, tests, pattern):
        if pattern is None:
            return tests
        return TestSuite(tests,
loader.discover(os.path.dirname(os.path.abspath(__file__)), pattern))

(The loader stores the top level directory it was originally called with
specifically for this use case. load_tests should not pass in a new top
level directory to the existing loader but create a new loader if it
wants to do this.)

Discovery does not follow the __path__ attribute of packages / modules
and only looks at the filesystem.

I have tested this implementation on the importlib tests (as one
example) and it worked fine.

Many of the restrictions mentioned here would be very easy to solve in
the future, but I think it is important to get some simple version of
test discovery in ASAP.

All I can think of for now...

[voidspace]

The usage information in TestProgram in this patch is not correct.

If __name__ != __main__ it should be unchanged - the new usage message
should only be displayed if unittest is run as __main__.

[voidspace]

Updated patch with documentation and fixed command line usage message.

Unless there are objections I intend to check this in in the next few
days. It would be helpful if someone else could go over the
documentation and check for errors / typos etc.

I've already had feedback from a few folks. The only suggestions so far
have been:

• Instead of a discover method the functionality could be built into
  TestLoader.loadTestsFromPath. This would be a substantial change and I
  think it better belongs in a new method.
• discover could simply load all modules and discover TestCases instead
  of needing a pattern to filter on. I think filtering is an important
  feature (not least for performance and because importing arbitrary
  modules is not safe). Loading all modules can be had by passing in a
  pattern of '*'

The behavior as implemented is a subset of the test discovery provided
by frameworks like nose (which doesn't require all test modules to be
importable from the top level).

The load_tests protocol is an idea already in use by the Bzr test
framework and similar protocols are in use in other frameworks.

[voidspace]

Georg Brandl has pointed out various minor issues in the docs (and
discover docstring) that I am correcting.

[pitrou]

Would you like to upload your patch to Rietveld?

[voidspace]

http://codereview.appspot.com/63157

[voidspace]

Committed in revision 73027.