unittest loader barfs on symlinks

[pitrou]

BPO	19352
Nosy	@warsaw, @doko42, @pitrou, @voidspace, @iritkatriel
Files	unittest_loader_symlinks.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-14.23:29:01.244>
created_at = <Date 2013-10-22.14:32:55.472>
labels = ['type-bug', 'library']
title = 'unittest loader barfs on symlinks'
updated_at = <Date 2021-08-14.23:29:01.243>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2021-08-14.23:29:01.243>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-14.23:29:01.244>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2013-10-22.14:32:55.472>
creator = 'pitrou'
dependencies = []
files = ['32304']
hgrepos = []
issue_num = 19352
keywords = ['patch']
message_count = 15.0
messages = ['200964', '200965', '200966', '201046', '201047', '201048', '204701', '204702', '204710', '204730', '204732', '204738', '204740', '204741', '399602']
nosy_count = 8.0
nosy_names = ['barry', 'doko', 'acapnotic', 'pitrou', 'michael.foord', 'python-dev', 'pitti', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19352'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[pitrou]

unittest.loader has the following snippet:

                    if realpath.lower() != fullpath_noext.lower():
                        module_dir = os.path.dirname(realpath)
                        mod_name = os.path.splitext(os.path.basename(full_path))[0]
                        expected_dir = os.path.dirname(full_path)
                        msg = ("%r module incorrectly imported from %r. Expected %r. "
                               "Is this module globally installed?")

Unfortunately, this will break with virtualenv on Ubuntu, which creates
a "local" directory full of symlinks. You end with this kind of error message:

======================================================================
ERROR: __main__ (unittest.loader.LoadTestsFailure)
----------------------------------------------------------------------
ImportError: 'test_asyncagi' module incorrectly imported from '/home/antoine/obelus/.tox/py27/local/lib/python2.7/site-packages/obelus/test'. Expected '/home/antoine/obelus/.tox/py27/lib/python2.7/site-packages/obelus/test'. Is this module globally installed?

Instead of (rather stupidly) calling lower(), realpath() and normcase()
should be called instead, to make sure the canonical paths are compared.

[pitrou]

Attaching patch.

[voidspace]

Good catch and fix Antoine. Thanks.

[python-dev]

New changeset d7ec961cea1c by Antoine Pitrou in branch '2.7':
Issue bpo-19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/d7ec961cea1c

[python-dev]

New changeset a830cc1c0565 by Antoine Pitrou in branch '3.3':
Issue bpo-19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/a830cc1c0565

New changeset ebbe87204114 by Antoine Pitrou in branch 'default':
Issue bpo-19352: Fix unittest discovery when a module can be reached through several paths (e.g. under Debian/Ubuntu with virtualenv).
http://hg.python.org/cpython/rev/ebbe87204114

[pitrou]

Fixed!

[doko42]

re-opening. the patch did break autopilot running with 2.7 on Ubuntu. I don't yet understand what this patch is supposed to fix.

[doko42]

see https://launchpad.net/bugs/1255505

[pitti]

More precisely, it broke unittest's discovery (not specific to autopilot). For any installed test, you now get:

$ python -m unittest discover lazr
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 113, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python2.7/unittest/main.py", line 214, in _do_discovery
    self.test = loader.discover(start_dir, pattern, top_level_dir)
  File "/usr/lib/python2.7/unittest/loader.py", line 206, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib/python2.7/unittest/loader.py", line 287, in _find_tests
    for test in self._find_tests(full_path, pattern):
  File "/usr/lib/python2.7/unittest/loader.py", line 287, in _find_tests
    for test in self._find_tests(full_path, pattern):
  File "/usr/lib/python2.7/unittest/loader.py", line 267, in _find_tests
    raise ImportError(msg % (mod_name, module_dir, expected_dir))
ImportError: 'test_error' module incorrectly imported from '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Expected '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Is this module globally installed?

I reverted this patch in Ubuntu for now as a quickfix.

This might be specific how Debian installs Python 2 modules. Packages ship them in /usr/share/pyshared/<modulename>/, and for every supported 2.X version, ship /usr/lib/python2.X/dist-packages/<modulename>/ which contains only the directories and *.pyc files, but symlinks the *.py files to /usr/share/pyshared/<modulename>/../*.py

So, it might be that upstream does not support this symlink layout, but it's the best thing to avoid having to install multiple *.py copies (this is all solved in a much more elegant way with Python 3, of course). But it would be nice if unittest's discovery could still cope with this.

Thanks!

[pitrou]

ImportError: 'test_error' module incorrectly imported from '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Expected '/usr/lib/python2.7/dist-packages/lazr/restfulclient/tests'. Is this module globally installed?

Well, this looks like the same path to me. Can you investigate a bit?

[voidspace]

This can happen when the code used to compare the paths is different from the code used to generate the failure message. This of course masks the real problem. I can look at where that might be possible and then hopefully we can get a genuine error message telling us the problem.

[pitti]

In this new code:

                    mod_file = os.path.abspath(getattr(module, '__file__', full_path))
                    realpath = os.path.splitext(os.path.realpath(mod_file))[0]
                    fullpath_noext = os.path.splitext(os.path.realpath(full_path))[0]

we get

modfile == /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error.pyc
realpath == /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error
fullpath_noext == /usr/share/pyshared/lazr/restfulclient/tests/test_error

for this file:

lrwxrwxrwx 1 root root 71 Mai 26 2013 /usr/lib/python2.7/dist-packages/lazr/restfulclient/tests/test_error.py -> ../../../../../../share/pyshared/lazr/restfulclient/tests/test_error.py

Which is as expected in Debian/Ubuntu as the *.pyc file is a real file in /usr/lib/python2.7, but the *.py is symlinked to /usr/share/.

This new patch essentially enforces that the *.py file is not a symlink, which breaks the Debian-ish way of installing python 2 modules.

[pitrou]

On ven., 2013-11-29 at 16:23 +0000, Martin Pitt wrote:

This new patch essentially enforces that the *.py file is not a
symlink, which breaks the Debian-ish way of installing python 2
modules.

So it doesn't help that Debian/Ubuntu likes to put symlinks everywhere,
then... (the original issue is due to another peculiarity you've added)

So, how about the following algorithm:

• check that both paths are equal
• if they aren't, call realpath() and check again
• if they are still unequal, raise an error

I suppose this is 2.7-only, btw? In 3.x, __file__ points to the py file
and not the pyc file.

[pitti]

Yes, this affects python 2.7 only; as I said, this is all solved in python3 by introducing the explicit extensions like __pycache__/*..cpython-33.pyc so that multiple versions can share one directory. With that these symlink hacks aren't necessary any more.

[iritkatriel]

this affects python 2.7 only;