ImportMismatchError from conftest.py confusion

[ckastner]

With a recent release of scikit-learn, we have started to see the following error when building packages for Debian using pytest 4.6.9 with Python 3.8 on Debian unstable:

/usr/lib/python3/dist-packages/_pytest/config/__init__.py:440: in _importconftest
    return self._conftestpath2mod[conftestpath]
E   KeyError: local('/build/scikit-learn-0.23.0/sklearn/conftest.py')

The package build happens roughly as follows:

1. python3.8 setup.py config
2. python3.8 setup.py build with a target directory .pybuild/cpython3_3.8/build/sklearn, which is a subdirectory within the source directory
3. cd .pybuild/cpython3_3.8/build/sklearn && python3.8 -m pytest

After step (2.), find . -name conftest.py in the source directory produces the following output (sklearn being the source directory, .pybuild the build directory:

./doc/conftest.py
./.pybuild/cpython3_3.8/build/sklearn/conftest.py
./.pybuild/cpython3_3.8/build/sklearn/externals/conftest.py
./.pybuild/cpython3_3.8/build/sklearn/utils/tests/conftest.py
./.pybuild/cpython3_3.8/build/sklearn/datasets/tests/conftest.py
./sklearn/conftest.py
./sklearn/externals/conftest.py
./sklearn/utils/tests/conftest.py
./sklearn/datasets/tests/conftest.py

Running step (3.) produces the following output:

===================================================== test session starts =====================================================
platform linux -- Python 3.8.3, pytest-4.6.9, py-1.8.1, pluggy-0.13.0 -- /usr/bin/python3.8
cachedir: .pytest_cache
rootdir: /build/scikit-learn-0.23.0, inifile: setup.cfg
collected 0 items / 1 errors                                                                                                  

=========================================================== ERRORS ============================================================
________________________________________________ ERROR collecting test session ________________________________________________
/usr/lib/python3/dist-packages/_pytest/config/__init__.py:440: in _importconftest
    return self._conftestpath2mod[conftestpath]
E   KeyError: local('/build/scikit-learn-0.23.0/sklearn/conftest.py')

During handling of the above exception, another exception occurred:
/usr/lib/python3/dist-packages/_pytest/config/__init__.py:446: in _importconftest
    mod = conftestpath.pyimport()
/usr/lib/python3/dist-packages/py/_path/local.py:721: in pyimport
    raise self.ImportMismatchError(modname, modfile, self)
E   py._path.local.LocalPath.ImportMismatchError: ('sklearn.conftest', '/build/scikit-learn-0.23.0/.pybuild/cpython3_3.8/build/sklearn/conftest.py', local('/build/scikit-learn-0.23.0/sklearn/conftest.py'))

During handling of the above exception, another exception occurred:
/usr/lib/python3/dist-packages/py/_path/common.py:383: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
/usr/lib/python3/dist-packages/py/_path/common.py:435: in gen
    for p in self.gen(subdir):
/usr/lib/python3/dist-packages/py/_path/common.py:435: in gen
    for p in self.gen(subdir):
/usr/lib/python3/dist-packages/py/_path/common.py:424: in gen
    dirs = self.optsort([p for p in entries
/usr/lib/python3/dist-packages/py/_path/common.py:425: in <listcomp>
    if p.check(dir=1) and (rec is None or rec(p))])
/usr/lib/python3/dist-packages/_pytest/main.py:667: in _recurse
    ihook = self.gethookproxy(dirpath)
/usr/lib/python3/dist-packages/_pytest/main.py:482: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
/usr/lib/python3/dist-packages/_pytest/config/__init__.py:424: in _getconftestmodules
    mod = self._importconftest(conftestpath.realpath())
/usr/lib/python3/dist-packages/_pytest/config/__init__.py:463: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info())
E   _pytest.config.ConftestImportFailure: (local('/build/scikit-learn-0.23.0/sklearn/conftest.py'), (<class 'py._path.local.LocalPath.ImportMismatchError'>, ImportMismatchError('sklearn.conftest', '/build/scikit-learn-0.23.0/.pybuild/cpython3_3.8/build/sklearn/conftest.py', local('/build/scikit-learn-0.23.0/sklearn/conftest.py')), <traceback object at 0x7f989924cc00>))

I saw that rootdir pointed to the source directory, and not the build directory (the cwd at this time). I assumed this was because the only copy of setup.cfg resided there. I copied that file to the build directory and ran pytest with --rootdir=$PWD, but that didn't change the outcome.

Oddly enough, removing the following two files from the source directory (not the build directory) resolves the issue.

./sklearn/conftest.py
./sklearn/datasets/tests/conftest.py

The other two conftest.py files in the source directory did not affect the outcome.

What could be going on here?

[asottile]

the KeyError is just the first in a multi stacktrace (due to some caching code) -- the actual error is the ImportMismatchError

[ckastner]

Good point, thanks. Retitled.

[nicoddemus]

Heh by coincidence, I just opened a PR in that sense yesterday: #7222. 😁

[ckastner]

I'm not sure as to the effects of that PR:

This removes the KeyError traceback from the chain when an conftest fails to import:

That sounds like adjusting the traceback, rather than addressing the issue that caused the ImportMismatchError in the first place.

Is this just a misunderstanding on my part?

[asottile]

oh, the ImportMismatchError is a user error -- it means you have two different paths which result to the same module (in this case, both the root conftest and the one inside your build directory are being picked up "identically")

[ckastner]

OK, so that's clear now. But what is the proper/recommended resolution for this?

From the docs, I would have thought that using --rootdir=<build directory> would have resolved this ambiguity, but the ImportMismatchError comes even when using that argument.

[asottile]

you have two separate copies of sklearn on the path. I'm not sure how to fix that without knowing the intricate details of your build but the solution is to not do that. It's a little weird to discover tests from an ephemeral build dir 🤔

[ckastner]

you have two separate copies of sklearn on the path. I'm not sure how to fix that without knowing the intricate details of your build but the solution is to not do that.

OK, that clarifies it somewhat, thanks. But this is pytest's own discovery, right?

It's a little weird to discover tests from an ephemeral build dir thinking

It's standard practice in Debian. We use ephemeral build dirs for various reasons, for example to build for more than one version of Python (3.7 and 3.8), hence we also run tests for both versions.

I'll check with the Debian Python Team, perhaps there is another solution to this problem that has escaped me so far.

Edit: Removed a sentence that was not relevant to ephemeral build dirs after all

[ckastner]

you have two separate copies of sklearn on the path

Wait, with "path", you mean one of pytest's own internal paths (for discovery or some such), right?

Because the source dir is definitely not in sys.path, and the build dir gets added by python3 -m pytest, if I read the docs correctly.