dvc modules shouldn't be named test_XXX.py

[rlamy]

Bug Report

Description

Modules named test_XXX.py have a special meaning for pytest and are handled as test files, which causes some weird side-effects with our dvc.testing.test_XXX modules:

• A minor one is that they get collected as test files if you run pytest dvc even though they're meant to run that way.
• More annoying: whenever they're imported from a test (even in a different project), pytest creates cache files next to them, potentially polluting site-packages with files that pip uninstall won't remove.

Reproduce

In a fresh virtualenv:

(dvc-s3) ronan@Creizic iterative % cd dvc-s3
(dvc-s3) ronan@Creizic dvc-s3 % pip install -e ".[tests]"
...
(dvc-s3) ronan@Creizic dvc-s3 % pytest dvc_s3
<SUCCESS>
(dvc-s3) ronan@Creizic dvc-s3 % ls /Users/ronan/.pyenv/versions/dvc-s3/lib/python3.8/site-packages/dvc/testing/__pycache__
__init__.cpython-38.pyc                    test_api.cpython-38.pyc
cloud.cpython-38.pyc                       test_remote.cpython-38-pytest-6.2.5.pyc
conftest.cpython-38.pyc                    test_remote.cpython-38.pyc
fixtures.cpython-38.pyc                    test_workspace.cpython-38-pytest-6.2.5.pyc
path_info.cpython-38.pyc                   test_workspace.cpython-38.pyc
test_api.cpython-38-pytest-6.2.5.pyc       tmp_dir.cpython-38.pyc
(dvc-s3) ronan@Creizic dvc-s3 % cd ../dvc
(dvc-s3) ronan@Creizic dvc % pip install -e .
...
(dvc-s3) ronan@Creizic dvc % cd -
(dvc-s3) ronan@Creizic dvc-s3 % ls /Users/ronan/.pyenv/versions/dvc-s3/lib/python3.8/site-packages/dvc/testing/__pycache__
test_api.cpython-38-pytest-6.2.5.pyc       test_workspace.cpython-38-pytest-6.2.5.pyc
test_remote.cpython-38-pytest-6.2.5.pyc
(dvc-s3) ronan@Creizic dvc-s3 % pytest dvc_s3
...
FAILED dvc_s3/tests/test_dvc.py::TestImport::test_import - AttributeError: module 'logging' has no attribute 'TRACE'
...

Note: the test failures happen because SITE-PACKAGES/dvc still exists (containing only the pytest cache files), which turns dvc into a namespace package, which causes dvc.__init__ not to be executed when importing dvc.

[efiop]

Just curious, how do packages that have tests in their core codebase deal with these problems?

[rlamy]

Good question! I don't know.

I guess it only causes issues when you switch from a regular to an editable installs, and you import the installed module (not the local directory). So it won't happen in CI, or with tox, because they install only once, and don't use -e. And it won't happen locally, because you're importing from local files. You have to be doing something a bit weird to hit this issue.

[efiop]

Gotcha. Alright, looks like we import these explicitly anyway, so this is likely more correct.