Feedback about --import-mode=importlib in pytest 6.0

[nicoddemus]

This issue is a placeholder to gather feedback from users about the new --import-mode=importlib introduced in pytest 6.0.

It is our intention to change the default to importlib in future versions, and perhaps even eliminate the other --import-mode options (prepend and append).

[The-Compiler]

It looks like the pytest-bdd plugin is incompatible with --import-mode=importlib:

_______________________________ ERROR collecting tests/end2end/features/test_backforward_bdd.py _______________________________
tests/end2end/features/test_backforward_bdd.py:21: in <module>
    bdd.scenarios('backforward.feature')
.tox/py38-pyqt515/lib/python3.8/site-packages/pytest_bdd/scenario.py:296: in scenarios
    features_base_dir = get_features_base_dir(module)
.tox/py38-pyqt515/lib/python3.8/site-packages/pytest_bdd/scenario.py:241: in get_features_base_dir
    default_base_dir = os.path.dirname(caller_module.__file__)
E   AttributeError: 'NoneType' object has no attribute '__file__'

Relevant code:

def scenarios(*feature_paths, **kwargs):
    frame = inspect.stack()[1]
    module = inspect.getmodule(frame[0])    # <- this is None

    features_base_dir = kwargs.get("features_base_dir")
    if features_base_dir is None:
        features_base_dir = get_features_base_dir(module)

Any idea why this could be happening?

[nicoddemus]

I think bdd.scenarios assumes the module given as arguments are available in sys.path, which is not the case when --import-mode=importlib.

Perhaps this will work (untested):

  module = inspect.getmodule(frame[0])
+ if module is None:
+     module = __import__(frame[0])

[schperplata]

Maybe this issue should be transfered here?

[nicoddemus]

Thanks @schperplata, but it seems that's a different issue however (I replied there).

[graingert]

refs: pytest-dev/py#203 (comment)

for me relative imports are broken when using --import-mode=importlib https://github.com/graingert/relative-imports-import-mode-importlib/blob/default/tests/test_foo.py#L1

============================================================================================================================== test session starts ===============================================================================================================================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/graingert/projects/relative-imports-import-mode-importlib, configfile: pytest.ini
plugins: forked-1.3.0, cov-2.9.0, xdist-1.34.0, case-1.5.3, timeout-1.4.2
collected 0 items / 1 error                                                                                                                                                                                                                                                      

===================================================================================================================================== ERRORS =====================================================================================================================================
_______________________________________________________________________________________________________________________ ERROR collecting tests/test_foo.py _______________________________________________________________________________________________________________________
ImportError while importing test module '/home/graingert/projects/relative-imports-import-mode-importlib/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_foo.py:1: in <module>
    from . import utils
E   ImportError: attempted relative import with no known parent package

when doing inline tests, I often use relative imports:

https://github.com/graingert/relative-imports-import-mode-importlib/blob/default/graingertspkg/tests/test_business_logic.py#L1

ImportError while importing test module '/home/graingert/projects/relative-imports-import-mode-importlib/graingertspkg/tests/test_business_logic.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
graingertspkg/tests/test_business_logic.py:1: in <module>
    from .. import business_logic
E   ImportError: attempted relative import with no known parent package

[nicoddemus]

for me relative imports are broken

Not sure they are broken, imports using importlib by design won't add modules to sys.modules or change sys.path, which explains why it doesn't work for your use case.

This however brings an important point, that importlib is not recommended for tests included with package code if one wants to use relative imports.

We should consider that as documentation, and also weight this use case in an eventual future decision of changing the default --importmode.

[graingert]

@nicoddemus would an --importmode=pkgutil that uses pkgutil.resolve_name for tests work with relative imports?

[nicoddemus]

Don't know, but should be easy to try, the logic is implemented in:

pytest/src/_pytest/pathlib.py

Line 445 in 91dbdb6

def import_path(

But one question is: what are you trying to fix by using a different import mode, and would that be fixed by using pkgutil.resolve_name?

[graingert]

I guess something like this, would work for inline tests:

    if mode is ImportMode.append:
        if str(pkg_root) not in sys.path:
            sys.path.append(str(pkg_root))
    elif mode is ImportMode.prepend:
        if str(pkg_root) != sys.path[0]:
            sys.path.insert(0, str(pkg_root))
    elif mode is ImportMode.manual:
        return importlib.import_module(module_name)

    importlib.import_module(module_name)

[graingert]

@nicoddemus maybe coupled with a collect mode that uses https://docs.python.org/3/library/importlib.html#importlib.resources.contents

so you can run with pytest -m tests

[nicoddemus]

Before any changes, the objective of the changes should be clear:

What are you trying to fix by using a different import mode, and would that be fixed by using pkgutil.resolve_name?

[graingert]

@nicoddemus ah sorry I realise now that pkgutil.resolve_name is a red-herring, after seeing the import_mode code.

For me the different import mode would be one that doesn't touch the sys.path and could be used if import-mode=prepend or append were removed, eg continuing to set __package__ for test modules but without any path manipulation.

When writing tests, (and any other python code), I think in terms of python -m and dotted package.module:attribute paths. I'd like an option for collect, nodeid and and import-mode that reflects this. eg collect would use https://docs.python.org/3/library/importlib.html#importlib.resources.contents to recursively iterate packages of all types (PEP420, PEP273, etc). nodeid would reflect the resource name/__qualname__ of the test function, and the import-mode would not touch sys.path. Pytest would call test functions is if they were run with pkgutil.resolve_name(nodeid)()

[nicoddemus]

Thanks!

Not sure how to accomplish this, perhaps @asottile can chime in.

[graingert]

@nicoddemus here's a demo of what I think I want: https://github.com/graingert/pieskytest

[asottile]

I think the subtlety we'd need to "break" in pytest is that pytest currently hoists all of the test modules to a flat level, but python devs expect folders in the tests dir to be actual modules and packages.

I'd love to put together something concrete about how we could opt into a world where pytest is less special about how it handles modules, but I fear this will break a lot that expect the old way (simply because it's been that way for a decade and code has been written against it)

an aside: iterating PEP420 modules is particularly tricky since technically an empty directory (or for instance ./docs in the pytest checkout) is a valid PEP420 "namespace package". pytest already has its own disovery code based upon folder-packages (folders not containing __init__.py) which is probably fine to use here, though it'd have to be adjusted to assume some sort of module root (and maybe have a setting for it? or just assume only sys.path roots?) -- this is another tricky part about this (relatedly why touching tests/__init__.py is a common practice)

[asottile]

I suspect the minimum to "fix" --mode=importlib is carefully setting __package__

[nicoddemus]

I think the subtlety we'd need to "break" in pytest is that pytest currently hoists all of the test modules to a flat level, but python devs expect folders in the tests dir to be actual modules and packages.

You mean with --importlib specifically or with other modes as well? If with --importlib you are correct, but for other modes that's not the case if the test module is located inside proper packages with __init__.py files, unless I'm missing something.

I suspect the minimum to "fix" --mode=importlib is carefully setting package

Nice. 👍 I think this can be tried without much risk: I suspect --mode=importlib is not widely used, and setting __package__ should not break those who use it today with success.

[graingert]

I think there may be confusion if users import conftest.py or import from other tests, which was always inadvisable, and it is a convenient side effect of --mode=importlib, that it's tricker to do

[yoav-orca]

I think maybe this is a better thread to share the issues I had with --import-mode=importlib #7859 & #7856

[maurosilber]

I had issues with code using get_type_hints. It can be reduced to this example:

from typing import get_type_hints

def test_type_hints():
    class A:
        x: int

    assert get_type_hints(A) == {"x": int}

It fails with a KeyError when trying to look up the module that defines test_type_hints() in sys.modules. I think it can be solved by #7870.

[CAM-Gerlach]

Thanks @nicoddemus for your thoughtful response and opening the issue! I'll reply further over there.

[bennyrowland]

@nicoddemus - since you seem to be the person with most of the answers: how do I use importlib with source code checking plugins? There are many plugins for pytest like pytest-flakes/pylint/black etc. which all add tests to read the package source code and run different checks on it. If I have a very simple package with a foo.py and a __init__.py containing import .foo then when pytest runs on these files with importlib then it fails to import them, and I get a huge stacktrace on stderr finishing with the "ImportError: attempted relative import with no known parent package" message. I understand that that is expected, but the file doesn't need to be imported, just passed to the various plugins. If I don't use any plugins, then there is no issue, pytest presumably tries importing the file, fails and moves on. So how to allow it to feed plugins without importing the file?

[The-Compiler]

@bennyrowland You might want to consider using a tool better suited to running code checking tools than pytest is (e.g. pre-commit or tox).

[nicoddemus]

So how to allow it to feed plugins without importing the file?

Hmm sorry I'm not sure I understand the relation to that regarding --import-mode...

[CAM-Gerlach]

@bennyrowland I'm not a Pytest dev, but as a basic check, you might want to confirm that the problem disappears by just changing the import-mode? If so, I imagine more precise detail on the exact structure of your project, both tests and code under test, and the full traceback and error message, would be helpful (and ideally, a minimal example repo).

And yeah, I've never really understood the desire for running any of these static checks with Pytest (as opposed to Pre-Commit if you want a check runner and/or tox/nox/etc. if you want unified orchestration), just like you wouldn't run Pytest under Pre-Commit; there's a pretty clear conceptual and practical separation between the needs and goals of static checks vs dynamic tests. But I digress...

[bennyrowland]

@The-Compiler, to be clear, these plugins are not my project, I am just a user. There are many plugins like: pytest-flake8, pytest-flakes, pytest-pylint, pytest-black, pytest-mypy etc. that all work in this way currently, and I understand are used by a good number of people. Changing to --import-mode=importlib will break all those tools for any repo that uses relative imports.

@CAM-Gerlach: I don't think that in this case an example repo is necessary - the code I provided in my original post is all that is necessary to break any of the above tools. Explicitly if pytest uses importlib to try to import a single source file that uses a relative import, it will fail by design. @nicoddemus points out earlier in this thread that that is (probably) correct behaviour for the test folder, but not for the main source tree. You are right that there are (possibly better) alternative ways to run these checkers and linters: I have found doing things this way to be convenient, but I will look into replacing things with pre-commit (which I do already use) and tox/nox (which I keep meaning to). However, all these plugin tools have worked correctly for a long time, a lot of people are using them and like the paradigm (as new plugins keep on proliferating for additional tools), it seems quite tough to just break the whole approach.

@nicoddemus, I think what I meant was this: using --import-mode=importlib is going to make importing source files for many packages impossible when they worked ok before. However, the tests that are being run on the source files (like mypy, pylint etc.) do not actually require those files to be imported, just passed to the external tools for processing. Therefore it would be possible to make all these plugins compatible with --importlib if the plugins could collect their test lists without the files having to be imported at all. I suspect that this may even be possible already, although none of the plugins I use regularly, or the others that I tried in encountering this issue worked in this case. So my question for you (and anybody else who can help) is whether this is something that needs to be fixed on the plugin side or on the pytest side in order for this to "just work" again.

[asottile]

can you provide a minimal example? I'm unable to reproduce

[bennyrowland]

@asottile here is an example repo: https://github.com/bennyrowland/pytest_plugin_example. Python 3.9 in a brand new env, install requirements.txt (only pytest-mypy) and run pytest --mypy --import-mode=importlib and I get the following traceback

H:\Python\pytest_plugin_example>pytest --import-mode=importlib --mypy
======================================================================================================== test session starts ========================================================================================================
platform win32 -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: H:\Python\pytest_plugin_example
plugins: mypy-0.8.1
collected 3 items                                                                                                                                                                                                                    

src\example\__init__.py ..                                                                                                                                                                                                     [ 66%]
src\example\example.py E                                                                                                                                                                                                       [100%]

============================================================================================================== ERRORS ===============================================================================================================
_____________________________________________________________________________________________ ERROR at setup of src\example\example.py ______________________________________________________________________________________________

self = <Package example>

    def _importtestmodule(self):
        # We assume we are only called once per module.
        importmode = self.config.getoption("--import-mode")
        try:
>           mod = import_path(self.fspath, mode=importmode)

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\python.py:578:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

p = local('H:\\Python\\pytest_plugin_example\\src\\example\\__init__.py')

    def import_path(
        p: Union[str, py.path.local, Path],
        *,
        mode: Union[str, ImportMode] = ImportMode.prepend,
    ) -> ModuleType:
        """Import and return a module from the given path, which can be a file (a module) or
        a directory (a package).

        The import mechanism used is controlled by the `mode` parameter:

        * `mode == ImportMode.prepend`: the directory containing the module (or package, taking
          `__init__.py` files into account) will be put at the *start* of `sys.path` before
          being imported with `__import__.

        * `mode == ImportMode.append`: same as `prepend`, but the directory will be appended
          to the end of `sys.path`, if not already in `sys.path`.

        * `mode == ImportMode.importlib`: uses more fine control mechanisms provided by `importlib`
          to import the module, which avoids having to use `__import__` and muck with `sys.path`
          at all. It effectively allows having same-named test modules in different places.

        :raises ImportPathMismatchError:
            If after importing the given `path` and the module `__file__`
            are different. Only raised in `prepend` and `append` modes.
        """
        mode = ImportMode(mode)

        path = Path(str(p))

        if not path.exists():
            raise ImportError(path)

        if mode is ImportMode.importlib:
            module_name = path.stem

            for meta_importer in sys.meta_path:
                spec = meta_importer.find_spec(module_name, [str(path.parent)])
                if spec is not None:
                    break
            else:
                spec = importlib.util.spec_from_file_location(module_name, str(path))

            if spec is None:
                raise ImportError(
                    "Can't find module {} at location {}".format(module_name, str(path))
                )
            mod = importlib.util.module_from_spec(spec)
>           spec.loader.exec_module(mod)  # type: ignore[union-attr]

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\pathlib.py:498:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_frozen_importlib_external.SourceFileLoader object at 0x03A3F430>, module = <module '__init__' from 'H:\\Python\\pytest_plugin_example\\src\\example\\__init__.py'>

>   ???

<frozen importlib._bootstrap_external>:850:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

f = <built-in function exec>
args = (<code object <module> at 0x03A69840, file "H:\Python\pytest_plugin_example\src\example\__init__.py", line 1>, {'__bui...t__.cpython-39.pyc', '__doc__': None, '__file__': 'H:\\Python\\pytest_plugin_example\\src\\example\\__
init__.py', ...})
kwds = {}

>   ???

<frozen importlib._bootstrap>:228:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   from . import example
E   ImportError: attempted relative import with no known parent package

src\example\__init__.py:1: ImportError

The above exception was the direct cause of the following exception:

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x034B6898>, when = 'setup', reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\runner.py:311:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\runner.py:255:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_HookCaller 'pytest_runtest_setup'>, args = (), kwargs = {'item': <MypyFileItem mypy>}, argname = 'item', firstresult = False

    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()

        # This is written to avoid expensive operations when not needed.
        if self.spec:
            for argname in self.spec.argnames:
                if argname not in kwargs:
                    notincall = tuple(set(self.spec.argnames) - kwargs.keys())
                    warnings.warn(
                        "Argument(s) {} which are declared in the hookspec "
                        "can not be found in this hook call".format(notincall),
                        stacklevel=2,
                    )
                    break

            firstresult = self.spec.opts.get("firstresult")
        else:
            firstresult = False

>       return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\pluggy\_hooks.py:265:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.config.PytestPluginManager object at 0x0158E640>, hook_name = 'pytest_runtest_setup'
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from 'C:\\Users\\Hallmarq.BEN-LAPTOP\\anaconda3\\envs\\sy...ture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin object a
t 0x033BA448>>, ...]
kwargs = {'item': <MypyFileItem mypy>}, firstresult = False

    def _hookexec(self, hook_name, methods, kwargs, firstresult):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\pluggy\_manager.py:80:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hook_name = 'pytest_runtest_setup'
hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from 'C:\\Users\\Hallmarq.BEN-LAPTOP\\anaconda3\\envs\\sy...ture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin objec
t at 0x033BA448>>, ...]
caller_kwargs = {'item': <MypyFileItem mypy>}, firstresult = False

    def _multicall(hook_name, hook_impls, caller_kwargs, firstresult):
        """Execute a call into multiple python functions/methods and return the
        result(s).

        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    f"hook call must provide argument {argname!r}"
                                )

                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
                        res = hook_impl.function(*args)
                        if res is not None:
                            results.append(res)
                            if firstresult:  # halt further impl calls
                                break
            except BaseException:
                excinfo = sys.exc_info()
        finally:
            if firstresult:  # first result hooks return a single value
                outcome = _Result(results[0] if results else None, excinfo)
            else:
                outcome = _Result(results, excinfo)

            # run all wrapper post-yield blocks
            for gen in reversed(teardowns):
                try:
                    gen.send(outcome)
                    _raise_wrapfail(gen, "has second yield")
                except StopIteration:
                    pass

>           return outcome.get_result()

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\pluggy\_callers.py:60:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pluggy._result._Result object at 0x034AC940>

    def get_result(self):
        """Get the result(s) for this hook call.

        If the hook was marked as a ``firstresult`` only a single value
        will be returned otherwise a list of results.
        """
        __tracebackhide__ = True
        if self._excinfo is None:
            return self._result
        else:
            ex = self._excinfo
>           raise ex[1].with_traceback(ex[2])

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\pluggy\_result.py:60:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

hook_name = 'pytest_runtest_setup'
hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from 'C:\\Users\\Hallmarq.BEN-LAPTOP\\anaconda3\\envs\\sy...ture=None>>, <HookImpl plugin_name='logging-plugin', plugin=<_pytest.logging.LoggingPlugin objec
t at 0x033BA448>>, ...]
caller_kwargs = {'item': <MypyFileItem mypy>}, firstresult = False

    def _multicall(hook_name, hook_impls, caller_kwargs, firstresult):
        """Execute a call into multiple python functions/methods and return the
        result(s).

        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    f"hook call must provide argument {argname!r}"
                                )

                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
>                       res = hook_impl.function(*args)

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\pluggy\_callers.py:39:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

item = <MypyFileItem mypy>

    def pytest_runtest_setup(item: Item) -> None:
        _update_current_test_var(item, "setup")
>       item.session._setupstate.prepare(item)

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\runner.py:150:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.runner.SetupState object at 0x033BA850>, colitem = <MypyFileItem mypy>

    def prepare(self, colitem) -> None:
        """Setup objects along the collector chain to the test-method."""

        # Check if the last collection node has raised an error.
        for col in self.stack:
            if hasattr(col, "_prepare_exc"):
                exc = col._prepare_exc  # type: ignore[attr-defined]
                raise exc

        needed_collectors = colitem.listchain()
        for col in needed_collectors[len(self.stack) :]:
            self.stack.append(col)
            try:
                col.setup()
            except TEST_OUTCOME as e:
                col._prepare_exc = e  # type: ignore[attr-defined]
>               raise e

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\runner.py:452:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.runner.SetupState object at 0x033BA850>, colitem = <MypyFileItem mypy>

    def prepare(self, colitem) -> None:
        """Setup objects along the collector chain to the test-method."""

        # Check if the last collection node has raised an error.
        for col in self.stack:
            if hasattr(col, "_prepare_exc"):
                exc = col._prepare_exc  # type: ignore[attr-defined]
                raise exc

        needed_collectors = colitem.listchain()
        for col in needed_collectors[len(self.stack) :]:
            self.stack.append(col)
            try:
>               col.setup()

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\runner.py:449:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Package example>

    def setup(self) -> None:
        # Not using fixtures to call setup_module here because autouse fixtures
        # from packages are not called automatically (#4085).
        setup_module = _get_first_non_fixture_func(
>           self.obj, ("setUpModule", "setup_module")
        )

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\python.py:644:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Package example>

    @property
    def obj(self):
        """Underlying Python object."""
        obj = getattr(self, "_obj", None)
        if obj is None:
>           self._obj = obj = self._getobj()

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\python.py:291:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Package example>

    def _getobj(self):
>       return self._importtestmodule()

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\python.py:500:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Package example>

    def _importtestmodule(self):
        # We assume we are only called once per module.
        importmode = self.config.getoption("--import-mode")
        try:
            mod = import_path(self.fspath, mode=importmode)
        except SyntaxError as e:
            raise self.CollectError(
                ExceptionInfo.from_current().getrepr(style="short")
            ) from e
        except ImportPathMismatchError as e:
            raise self.CollectError(
                "import file mismatch:\n"
                "imported module %r has this __file__ attribute:\n"
                "  %s\n"
                "which is not the same as the test file we want to collect:\n"
                "  %s\n"
                "HINT: remove __pycache__ / .pyc files and/or use a "
                "unique basename for your test file modules" % e.args
            ) from e
        except ImportError as e:
            exc_info = ExceptionInfo.from_current()
            if self.config.getoption("verbose") < 2:
                exc_info.traceback = exc_info.traceback.filter(filter_traceback)
            exc_repr = (
                exc_info.getrepr(style="short")
                if exc_info.traceback
                else exc_info.exconly()
            )
            formatted_tb = str(exc_repr)
>           raise self.CollectError(
                "ImportError while importing test module '{fspath}'.\n"
                "Hint: make sure your test modules/packages have valid Python names.\n"
                "Traceback:\n"
                "{traceback}".format(fspath=self.fspath, traceback=formatted_tb)
            ) from e
E           _pytest.nodes.Collector.CollectError: ImportError while importing test module 'H:\Python\pytest_plugin_example\src\example\__init__.py'.
E           Hint: make sure your test modules/packages have valid Python names.
E           Traceback:
E           src\example\__init__.py:1: in <module>
E               from . import example
E           E   ImportError: attempted relative import with no known parent package

C:\Users\Hallmarq.BEN-LAPTOP\anaconda3\envs\symlink\lib\site-packages\_pytest\python.py:603: CollectError
=============================================================================================================== mypy ================================================================================================================
Success: no issues found in 2 source files
====================================================================================================== short test summary info ======================================================================================================
ERROR src/example/example.py::mypy - _pytest.nodes.Collector.CollectError: ImportError while importing test module 'H:\Python\pytest_plugin_example\src\example\__init__.py'.
==================================================================================================== 2 passed, 1 error in 0.61s =====================================================================================================

[asottile]

that points to a bug in pytest-mypy, it shouldn't be constructing Package / Module discovery items (it doesn't use them, just passes the filename along to mypy)

[bennyrowland]

@asottile, thanks, that is useful information, I figured it was at least a 50/50 that the issue was in the plugins. I will see if I can figure out how to fix the issue there, although it is worth pointing out that there are at least 6 pytest plugins of this nature that I tested and all have the same issue. That may be because they are all copying each other, but it may also be worth taking a look at pytest's own documentation on creating plugins as they are all going wrong.

[CAM-Gerlach]

At least naively as I understand it, the broader issue seems to be that these plugins are hooking into Pytest's test discovery system to feed them the files to analyze, when it at least nominally isn't really primarily designed for this use case. Given it seems the legacy import-modes will stick around indefinitely as a non-default option (at least per the discussion above), it should still be possible for plugins to do this if users pass the appropriate config option/CLI flag, if they don't want to migrate to alternative file-discovery methods.

While it may affect these types of plugins generally, I'm not sure if it would make sense to do anything further on the Pytest side, given this isn't really the intended use case of test collection, as opposed to either the aforementioned workarounds or recommending a more appropriate tool, most notably @asottile 's superb Pre-Commit (particularly since you're already using it).

Protip: If you surround your huge code block with <details> tags, it won't take up pages of space in the thread, and python-traceback will give you nice formatting:

<details>

```python-traceback
TRACEBACK HERE
```

</details>

[RolfBippus]

I actually have trouble both with import-mode importlib as well as prepend/append, neither of them doing what I expect:

Assume I have a project including a package as well as an installed version of the same package in my environment.
The package includes an integrated tests package.

project/python/mypackage
project/python/mypackage/tests
project/python/mypackage/tests/test_mypackage  # includes an import mypackage

install/mypackage
install/mypackage/tests
install/mypackage/tests/test_mypackage  # includes an import mypackage

Now, what shall I do when trying to

1. test the code in my project I am working on
2. test the installed code

Using importlib mode:

• Running plain pytest in my project folder, pytest is unable to find packages/modules in my project since none of the necessary paths have been added to sys.path. (ModuleNotFoundError: mypackage)
• Running pytest --pyargs my_package indeed runs the installed tests. Always.
• So to get plain pytest to work I tried to add paths manually to sys.path in pytest_configure() in conftest.py. But now pytest --pyargs mypackage executes project/python/mypackage/tests/test_mypackage if executed in the project folder.

Using prepend mode:

• Running plain pytest in my project folder correctly runs tests in my projectfolder.

• pytest --pyargs my_package runs tests either on the installed code or on the code in the project folder, depending on where I do actually execute it. This is slightly worse than the first option, since I cannot be sure what happens. E.g.

  • project/python > pytest --pyargs my_package will run project/python/mypackage/tests/test_mypackage
  • project > pytest --pyargs my_package will run install/mypackage/tests/test_mypackage

So all import modes do not really work consistently and transparently.

[CAM-Gerlach]

See the Pytest docs, which explains much of this.

• Running plain pytest in my project folder, pytest is unable to find packages/modules in my project since none of the necessary paths have been added to sys.path. (ModuleNotFoundError: mypackage)

Yes, this is exactly what you'd expect (and, generally, want) from Python's normal behavior, since you're using a de-facto src layout (just calling the directory python instead, I'm assuming due to having a substantial non-Python component to your project). This means that, since python is not itself a Python import package (nor should it be), the local non-installed copy of your mypackage package doesn't get picked up. This is a good thing, because it means you can confidently test your installed code (as you should, to ensure the package actually works when installed) without the local copy interfering, which is one of the main benefits of the src directory layout.

In order to test the local copy of your project, the standard best practice is just to install it in editable mode in your development environment (pip install -e .), assuming your packaging config (pyproject.toml, setup.cfg, etc) is set up correctly.

• Running pytest --pyargs my_package indeed runs the installed tests. Always.

Yes, as expected.

• So to get plain pytest to work I tried to add paths manually to sys.path in pytest_configure() in conftest.py. But now pytest --pyargs mypackage executes project/python/mypackage/tests/test_mypackage if executed in the project folder.

You should basically never, ever munge paths like this, as it is a path that leads to the dark side. What you're effectively doing, if I'm interpreting what you're saying correctly, is psuedo-"installing" the package, which the modes prepend and append do for you anyway, but in a way not directly controlled by Pytest, leading to --pyargs not being able to access the installed package, while not properly testing the actual package install, and potentially leading to other problems.

(Sidenote: If you're specifying CLI options or other config every time, which you generally are if you're running your tests rigorously, you can and should simply add it to your pytest.ini)

• Running plain pytest in my project folder correctly runs tests in my projectfolder.

It is fine for the tests to be run from your project folder, but the code under test should be from the installed package. To achieve that result, you should be able to do so with an editable install of your package, combined with the importlib mode.

• pytest --pyargs my_package runs tests either on the installed code or on the code in the project folder, depending on where I do actually execute it. This is slightly worse than the first option, since I cannot be sure what happens. E.g.

  • project/python > pytest --pyargs my_package will run project/python/mypackage/tests/test_mypackage
  • project > pytest --pyargs my_package will run install/mypackage/tests/test_mypackage

Yes, this is as expected, following the previous explanation. The current working directory gets added to sys.path, which means that per the above, your my_package package doesn't get picked up when you're in the project root directory (which you generally don't want), whereas it does if you happen to cd into your src (here named python) directory, since it is de-facto "installed" by being on the CWD path.

So all import modes do not really work consistently and transparently.

Some recommendations to ensure consistent behavior for your case:

• Use the importlib mode with addopts = import-mode=importlib in your pytest.ini
• To test your local code, editable-install your package in your development env with pip install -e .
• To test your fully installed package (i.e. installed from a wheel built with python -m build, PyPI, etc) activate an env with it installed.

[RolfBippus]

Wow, thanks for the thorough answer.

My misconception was, that for testing the development code I could "just run pytest", since with import mode prepend/append this is what (apparently) happens.
Since prepend/append however do not solve the ambiguity of which code is actually tested, I agree that importlib mode is better suited as it "forces" you to test on "properly installed packages" in order to get rid of the possible ambiguity. Possibly in editable mode if it shall be the development code.

Thanks for the clarification

[tobiasdiez]

I'm hitting a strange issue when trying to test files that are not part of the actual project. Assume that the project is located at /A/src with a file A/src/conftest.py and pytest config in A/src/tox.ini. With this setup, testing a file A/src/test.py works fine. However, when I copy the same file to a folder /B somewhere else and run pytest /B/test.py --impot-mode=importlib then this fails with the error message

ERROR: not found: /B/test.py
(no name '/B/test.py' in any of [])

Does someone has an idea whats going on?

[CAM-Gerlach]

Well, unless the directory B is at the root (/) of your drive, pytest is not going to find it at /B/test.py, importlib or not. Did you test that pytest /B/test.py --import-mode=prepend works?

Also, --impot-mode [sic] is not spelled correctly in the command invocation posted above.

Finally, to note, with this current configuration src is your top-level (import) package directory (rather than your package being a directory under src), which if you're developing a Python package (as opposed to support code for something else), is probably not what you want...

[nicoddemus]

Thanks everyone for the feedback.

Based on your feedback, it is clear that we should not change the importmode default to importlib, as it has its own set of drawbacks, not making it a clear winner, and changing the default would severely impact a lot of test suites.

Closing this issue then, feel free to open new issues to discuss specific topics related to --import-mode. 👍