__init__ of tests directory is imported twice if the parent is a namespace package

[sschwarzer]

Thank you for pytest! :-)

When running tests in a regular tests package inside a namespace package (see PEP 420), tests.__init__ is imported twice.

Setup:

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ cat /etc/fedora-release 
Fedora release 25 (Twenty Five)

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ python --version
Python 3.5.3

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ pytest --version
This is pytest version 3.1.2, imported from /home/schwa/.virtualenvs/pytest_test/lib/python3.5/site-packages/pytest.py

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ pip list --format=columns
Package    Version 
---------- --------
pip        9.0.1   
py         1.4.34  
pytest     3.1.2   
setuptools 36.0.1  
wheel      0.30.0a0

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ echo $PYTHONPATH 
/home/schwa/sandbox/pytest_test

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ tree
.
└── package
    └── tests
        ├── __init__.py
        └── test_foo.py

2 directories, 2 files

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ cat package/tests/__init__.py 
print("imported __init__")

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ cat package/tests/test_foo.py 
import package.tests


def test_foo():
    pass

Running pytest:

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ pytest -s package/tests
============================= test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/schwa/sandbox/pytest_test, inifile:
collecting 0 itemsimported __init__
imported __init__
collected 1 items 

package/tests/test_foo.py .

=========================== 1 passed in 0.01 seconds ===========================

Note that imported __init__ is printed twice. I expected it to be printed only once.

Indeed when I "convert" package to a regular Python package:

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ touch package/__init__.py

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ tree
.
└── package
    ├── __init__.py
    └── tests
        ├── __init__.py
        └── test_foo.py

2 directories, 3 files

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ pytest -s package/tests
============================= test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/schwa/sandbox/pytest_test, inifile:
collecting 0 itemsimported __init__
collected 1 items 

package/tests/test_foo.py .

=========================== 1 passed in 0.01 seconds ===========================

imported __init__ is only printed once.

Between the runs I removed the __pycache__ directories. This isn't shown here.

Of course, this is a simplified example. I think it applies to any case where you have one or more namespace package "above" one or more regular Python packages, which finally contain the tests package, i. e.

>= 1 namespace packages / >= 1 regular packages / `tests` package

In my project the above behavior leads to some configuration code being executed twice, which results in an exception. For now I can use the workaround of creating the __init__.py file in the should-be namespace package.

[sschwarzer]

Zip file with directory structure:

pytest_test.zip

[sschwarzer]

With package as a namespace package, tests is imported with different paths:

(pytest_test) schwa@warpy:~/sandbox/pytest_test$ cat package/tests/test_foo.py 
import sys

print("before importing `package.tests`, `package.test` in `sys.modules`",
      "package.tests" in sys.modules)
print("before importing `package.tests`", sys.modules["tests"])
print()

import package.tests

print("after importing `package.tests`", sys.modules["package.tests"])
print("after importing `package.tests`", sys.modules["tests"])


def test_foo():
    pass


(pytest_test) schwa@warpy:~/sandbox/pytest_test$ pytest -s package/tests
============================= test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/schwa/sandbox/pytest_test, inifile:
collecting 0 itemsimported __init__
before importing `package.tests`, `package.test` in `sys.modules` False
before importing `package.tests` <module 'tests' from '/home/schwa/sandbox/pytest_test/package/tests/__init__.py'>

imported __init__
after importing `package.tests` <module 'package.tests' from '/home/schwa/sandbox/pytest_test/package/tests/__init__.py'>
after importing `package.tests` <module 'tests' from '/home/schwa/sandbox/pytest_test/package/tests/__init__.py'>
collected 1 items 

package/tests/test_foo.py .

=========================== 1 passed in 0.01 seconds ===========================

[sschwarzer]

Meanwhile I found the notes under Tests as part of application code and think they may be related. I also found the pytest option --import-mode and tried --import-mode=append but that doesn't seem to help: tests/__init__.py is still imported under different names.

Is there a way to set the basedir for test modules explicitly? What do you recommend (ideally without modifying the directory structure or adding the __init__.py file to the "should-be" namespace package)?

[nicoddemus]

@sschwarzer sorry but the short answer is that pytest does not support namespace packages yet.

[dynofu]

also run into this problem when i use namespace package along with absl.flags which does allow one flag defined in 2 modules but because the "evolved import technique" one module is being imported twice with different names and hit this infamous absl.flags._exceptions.DuplicateFlagError.
is there a way to specify the basedir or disable this import technique?

[AlexanderMartynoff]

Hi everyone, is there any solution to the problem?