Test discovery finds module that doesn't exist

[jaraco]

In jaraco/inflect#135, the contributor has accidentally misnamed the package. They attempted to rename inflect (module) to inflect (package). Instead, by accident, they renamed inflect to inflect.inflect and as a result, created an empty inflect module (as a namespace package). I would have expected the tests to fail, but they did not. Something about the test collection caused pytest to find inflect/inflect.py as the top-level inflect module, masking the mistake.

troubleshooting

I tried to create a minimal reproducer, but the reproducer fails as expected:

draft $ tree
.
├── inflect
│   └── inflect.py
└── test_something.py

1 directory, 2 files
draft $ cat inflect/*
value = 'exists'
draft $ cat test*
import inflect


def test_something():
    assert inflect.value == 'exists'
draft $ pip-run pytest -- -m pytest
Collecting pytest
  Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting py>=1.8.2
  Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting packaging
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting attrs>=19.2.0
  Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB)
Installing collected packages: pyparsing, toml, py, pluggy, packaging, iniconfig, attrs, pytest
Successfully installed attrs-21.4.0 iniconfig-1.1.1 packaging-21.3 pluggy-1.0.0 py-1.11.0 pyparsing-3.0.7 pytest-6.2.5 toml-0.10.2
WARNING: You are using pip version 21.3.1; however, version 22.0.2 is available.
You should consider upgrading via the '/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m pip install --upgrade pip' command.
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                                 

test_something.py F                                                                                                                                        [100%]

============================================================================ FAILURES ============================================================================
_________________________________________________________________________ test_something _________________________________________________________________________

    def test_something():
>       assert inflect.value == 'exists'
E       AttributeError: module 'inflect' has no attribute 'value'

test_something.py:5: AttributeError
==================================================================== short test summary info =====================================================================
FAILED test_something.py::test_something - AttributeError: module 'inflect' has no attribute 'value'
======================================================================= 1 failed in 0.03s ========================================================================

I'm confident that one of the plugins (doctests, mypy, black, flake8, ...) is probably implicated, but when I try to enable doctests, I get another error:

draft $ pip-run -q pytest -- -m pytest --doctest-modules
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item / 1 error                                                                                                                                       

============================================================================= ERRORS =============================================================================
______________________________________________________________ ERROR collecting inflect/inflect.py _______________________________________________________________
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/runner.py:311: in from_call
    result: Optional[TResult] = func()
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/runner.py:341: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/doctest.py:532: in collect
    module = import_path(self.fspath)
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-q5qn9lt4/_pytest/pathlib.py:533: in import_path
    if module_file.endswith((".pyc", ".pyo")):
E   AttributeError: 'NoneType' object has no attribute 'endswith'
==================================================================== short test summary info =====================================================================
ERROR inflect/inflect.py - AttributeError: 'NoneType' object has no attribute 'endswith'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================== 1 error in 0.10s ========================================================================

It seems that error is occurring because inflect is discovered as a namespace package, but doesn't have a __file__ attribute.

I found that I could minimally reproduce the non-failure by (a) putting the test in a subdirectory and (b) invoking doctests.

draft $ tree
.
├── inflect
│   └── inflect.py
└── tests
    └── test_something.py

2 directories, 2 files
draft $ cat inflect/*
value = 'exists'
draft $ cat tests/*
import inflect


def test_something():
    assert inflect.value == 'exists'
draft $ pip-run -q pytest -- -m pytest --doctest-modules
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                                 

tests/test_something.py .                                                                                                                                  [100%]

======================================================================= 1 passed in 0.02s ========================================================================

As you can see, the test passes, even though the 'inflect' module should have only been found at inflect.inflect. Remove the --doctest-modules and the expected failure occurs:

draft $ pip-run -q pytest -- -m pytest
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/jaraco/draft
collected 1 item                                                                                                                                                 

tests/test_something.py F                                                                                                                                  [100%]

============================================================================ FAILURES ============================================================================
_________________________________________________________________________ test_something _________________________________________________________________________

    def test_something():
>       assert inflect.value == 'exists'
E       AttributeError: module 'inflect' has no attribute 'value'

tests/test_something.py:5: AttributeError
==================================================================== short test summary info =====================================================================
FAILED tests/test_something.py::test_something - AttributeError: module 'inflect' has no attribute 'value'
======================================================================= 1 failed in 0.03s ========================================================================

I'm fairly certain this missed failure is just another manifestation of #3396 and #8332. In this case, the namespace package was an accident.

[RonnyPfannschmidt]

Yeah, what happens is that the file gets found, and then pytest adds the packages own namespace folder to sys.path in order to get the doctests out of the file