Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test discovery for unittest #50251

voidspace opened this issue May 11, 2009 · 7 comments

Test discovery for unittest #50251

voidspace opened this issue May 11, 2009 · 7 comments
stdlib Python modules in the Lib dir type-feature A feature request or enhancement


Copy link

BPO 6001
Nosy @gpshead, @pitrou, @benjaminp, @voidspace
  • test_discovery.patch
  • test_discovery.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = ''
    closed_at = <Date 2009-05-29.20:44:19.885>
    created_at = <Date 2009-05-11.21:40:35.814>
    labels = ['type-feature', 'library']
    title = 'Test discovery for unittest'
    updated_at = <Date 2009-05-29.20:44:19.883>
    user = '' fields:

    activity = <Date 2009-05-29.20:44:19.883>
    actor = 'michael.foord'
    assignee = 'michael.foord'
    closed = True
    closed_date = <Date 2009-05-29.20:44:19.885>
    closer = 'michael.foord'
    components = ['Library (Lib)']
    creation = <Date 2009-05-11.21:40:35.814>
    creator = 'michael.foord'
    dependencies = []
    files = ['13961', '14052']
    hgrepos = []
    issue_num = 6001
    keywords = ['patch', 'needs review']
    message_count = 7.0
    messages = ['87591', '87616', '88268', '88271', '88434', '88448', '88536']
    nosy_count = 4.0
    nosy_names = ['gregory.p.smith', 'pitrou', 'benjamin.peterson', 'michael.foord']
    pr_nums = []
    priority = 'normal'
    resolution = 'accepted'
    stage = 'patch review'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = ''
    versions = ['Python 2.7', 'Python 3.2']

    Copy link
    Contributor Author

    Attached is a patch that implements test discovery for unittest.

    It includes command line argument handling (awkward manual handling but
    works fine...), so that it can be invoked through:

    python -m unittest discover
    python -m unittest discover start_dir pattern top_level_dir

    If the patch is acceptable then I will write docs. Do I need to make a
    post to Python-dev or is it ok to have it reviewed here? The basic
    mechanism was hammered out on the Testing-in-Python mailing list with a
    lot of input from Robert Collins.

    Brett Cannon has already looked over the test discovery code (not the
    command line handling code though!).

    It includes a customization hook both for modules and packages to
    customize test loading and discovery. Described below:

    Advantage of this: it is simple (easy to extend)
    Disadvantage: it is simple (a lot it doesn't do)

    Test discovery is provided through the method.

    It takes three arguments - start directory, pattern to match test files
    (defaults to 'test*.py') and the top level directory of the project
    (defaults to the start directory).

    The main restriction is that all your tests must be importable from the
    top level directory of your project. The start directory is then
    recursively searched for files and packages that match the pattern you
    pass in. Tests are loaded from matching modules, and all tests run.

    (A further restriction is that discovery only currently recognises
    packages as directories containing files and not .pyc etc.)

    The load_tests protocol provides a way for packages and modules to
    customize test loading. This is inspired by a similar system in use in
    the Bzr test suite.

    Iff a test module defines a load_tests function then
    TestLoader.loadTestsFromModule will call this function with loader,
    tests, None. This should return a suite.

    An example 'do nothing' implementation of load_tests would be:

       def load_tests(loader, tests, pattern):
           return tests

    If a package directory name matches the pattern you pass into discovery
    and the contains a load_tests function then it will be
    called with loader, tests, pattern. No further discovery will be done
    into the package, but because it is passed the pattern as an argument it
    is free to continue discovery itself. A do nothing load_tests for a
    package is:

        def load_tests(loader, tests, pattern):
            if pattern is None:
                return tests
            return TestSuite(tests,, pattern))

    (The loader stores the top level directory it was originally called with
    specifically for this use case. load_tests should not pass in a new top
    level directory to the existing loader but create a new loader if it
    wants to do this.)

    Discovery does not follow the __path__ attribute of packages / modules
    and only looks at the filesystem.

    I have tested this implementation on the importlib tests (as one
    example) and it worked fine.

    Many of the restrictions mentioned here would be very easy to solve in
    the future, but I think it is important to get some simple version of
    test discovery in ASAP.

    All I can think of for now...

    @voidspace voidspace self-assigned this May 11, 2009
    @voidspace voidspace added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 11, 2009
    Copy link
    Contributor Author

    The usage information in TestProgram in this patch is not correct.

    If __name__ != __main__ it should be unchanged - the new usage message
    should only be displayed if unittest is run as __main__.

    Copy link
    Contributor Author

    Updated patch with documentation and fixed command line usage message.

    Unless there are objections I intend to check this in in the next few
    days. It would be helpful if someone else could go over the
    documentation and check for errors / typos etc.

    I've already had feedback from a few folks. The only suggestions so far
    have been:

    • Instead of a discover method the functionality could be built into
      TestLoader.loadTestsFromPath. This would be a substantial change and I
      think it better belongs in a new method.
    • discover could simply load all modules and discover TestCases instead
      of needing a pattern to filter on. I think filtering is an important
      feature (not least for performance and because importing arbitrary
      modules is not safe). Loading all modules can be had by passing in a
      pattern of '*'

    The behavior as implemented is a subset of the test discovery provided
    by frameworks like nose (which doesn't require all test modules to be
    importable from the top level).

    The load_tests protocol is an idea already in use by the Bzr test
    framework and similar protocols are in use in other frameworks.

    Copy link
    Contributor Author

    Georg Brandl has pointed out various minor issues in the docs (and
    discover docstring) that I am correcting.

    Copy link

    pitrou commented May 27, 2009

    Would you like to upload your patch to Rietveld?

    Copy link
    Contributor Author

    Copy link
    Contributor Author

    Committed in revision 73027.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    None yet

    No branches or pull requests

    2 participants