Unittest discover fails with namespace package if the path contains the string same as the module name

[toshishigehagihara]

BPO	24168
Nosy	@warsaw, @rbtcollins, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-11.23:43:13.139>
created_at = <Date 2015-05-12.01:14:38.021>
labels = ['type-bug', 'tests']
title = 'Unittest discover fails with namespace package if the path contains the string same as the module name'
updated_at = <Date 2016-09-11.23:43:13.138>
user = 'https://bugs.python.org/toshishigehagihara'

bugs.python.org fields:

activity = <Date 2016-09-11.23:43:13.138>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-11.23:43:13.139>
closer = 'barry'
components = ['Tests']
creation = <Date 2015-05-12.01:14:38.021>
creator = 'toshishige hagihara'
dependencies = []
files = []
hgrepos = []
issue_num = 24168
keywords = []
message_count = 5.0
messages = ['242929', '242930', '242932', '244847', '275888']
nosy_count = 4.0
nosy_names = ['barry', 'rbcollins', 'eric.snow', 'toshishige hagihara']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24168'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[toshishigehagihara]

There is a problem with unittest discovering with namespace packages.
Given the following directory structure, the following command fails with the errors.

# Directory Structure

/home/hagihara/test.cybozu/infra/forest/lib/python3/
    cybozu/   # <- namespace package
        cmdb/
            __init__.py
            test.py

# Commands

# setup module path
echo /home/hagihara/test.cybozu/infra/forest/lib/python3 > sudo dd of=/usr/lib/python3/dist-packages/cybozu.pth
cd /home/hagihara/test.cybozu/infra
python3 -m unittest discover -s cybozu

# Errors

$ python3 -m unittest discover -s cybozu
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/loader.py", line 221, in discover
    os.path.dirname((the_module.__file__)))
AttributeError: 'module' object has no attribute '__file__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.4/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/usr/lib/python3.4/unittest/main.py", line 92, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python3.4/unittest/main.py", line 116, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python3.4/unittest/main.py", line 225, in _do_discovery
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/usr/lib/python3.4/unittest/loader.py", line 242, in discover
    namespace=True))
  File "/usr/lib/python3.4/unittest/loader.py", line 349, in _find_tests
    namespace=namespace)
  File "/usr/lib/python3.4/unittest/loader.py", line 310, in _find_tests
    name = self._get_name_from_path(full_path)
  File "/usr/lib/python3.4/unittest/loader.py", line 284, in _get_name_from_path
    assert not _relpath.startswith('..'), "Path must be within the project"
AssertionError: Path must be within the project

# Cause of the error

This error happens because TestLoader.discover does not set self._top_level_dir properly.

/usr/lib/python3.4/unittest/loader.py

class TestLoader(object):
    ...
    def discover(self, start_dir, pattern='test*.py', top_level_dir=None):
        ...
        if os.path.isdir(os.path.abspath(start_dir)):
            ...
        else:
           # support for discovery from dotted module names
            try:
                __import__(start_dir)
            except ImportError:
                is_not_importable = True
            else:
                try:
                    start_dir = os.path.abspath(
                       os.path.dirname((the_module.__file__)))
                except AttributeError:
                    # look for namespace packages
                    ...
                    if spec and spec.loader is None:
                        if spec.submodule_search_locations is not None:
                            is_namespace = True

                            for path in the_module.__path__:
                                if (not set_implicit_top and
                                    not path.startswith(top_level_dir)):
                                    continue
                                self._top_level_dir = \                    <--- cause of the error.
                                    (path.split(the_module.__name__
                                         .replace(".", os.path.sep))[0])

If path of the module contains the string same as the module name,
the path is split at incorrect position and invalid value is set to self._top_level_dir.

path.split(the_module.__name__.replace(".", os.path.sep))[0]

the_module -> cybozu
cybozu.__path__ -> ['/home/hagihara/test.cybozu/infra/forest/lib/python3/cybozu']
path.split('cybozu')[0] -> '/home/hagihara/test.'

[ericsnowcurrently]

Just in case, please take a look at issues bpo-17457 and bpo-23882 to see if they already cover the bug.

[toshishigehagihara]

Thanks for suggestion. I checked both issues and found that this bug is created in bpo-17457 and bpo-23882 does not fix it.

[warsaw]

If you need a test case, try https://gitlab.com/warsaw/flufl.lock

[warsaw]

I think this bug actually is fixed now. I've tried my previously failing case in Python 3.5 and 3.6 head and I don't get the error.