Warn user in case of "installed package" shadowed by "source package"

[pakal]

Hello,

I've crossed numerous times the dangerous "installed package shadowed by source package" bug as described in #514

In some rare cases (eg. if the code needed to be transformed with 2to3 in setup.py), this raised an error and I could fix my pythonpath accordingly.
But most of the time, the error was silent, so I ended up testing the source package, not the installed one, with a risk of pushing to Pypi an incomplete/broken package.

I know the pythonpath problem can't be solved by Tox alone (despite nice protections like blocking PYTHONPATH transfer by default), since many factors are at work here: python adding (or not) cwd() to sys.path, pytest/nose and its plugins adding directories as they are discovered (eg. those with a conftest.py in it), presence of sources at the root of repository or in a "src/" subfolder... so the Tox configuration has to make sure that the final test launches occur in an environment without unwanted sys.path folders. But I guess that users could be warned when their conf is broken.

Would it be possible to automatically issue a check after virtualenv/install steps, to force an import on the "installed package" (no idea what's the best way to determine its name), and raise an exception if the imported package is not located in the "site-packages/" directory of the Tox virtualenv?
Maybe this check could be silenced by users on demand (though I can't think of a case when installing outsie the virtualenv is expected...), but having this safety by default would be invaluable (for now I add my own pytest test to check that).

[gaborbernat]

There's no easy way to do this. The problem is that many people actually use the fact that the test runs on the source. We can encourage people to test what they install but don't see an easy way to force them to do that, and sometimes that might not be even possible; e.g. if pytest is invoked via -m the cwd is always added to the sys.path. So I don't think tox can do anything here 🤔 If anyone has good ideas let me know and we can reopen.

[pakal]

Newcomers might have the impression that Tox tests in an isolated environment, so projects here and there seem to discover "the hard way" that they were not actually testing the right package. Maybe some additional warnings in documentation homepage, and outputs during execution, so that the problem gets detected earlier ?