Running wdspec directly via pytest is unable to collect all tests

[gsnedders]

With:

(venv) gsnedders@gsnedders-marsha webdriver % export PYTHONPATH="$( pwd )/../tools/webdriver:$( pwd )/../tools/third_party/websockets/src:$( pwd ):$( pwd )/.."
(venv) gsnedders@gsnedders-marsha webdriver % pytest --collect-only --rootdir=.. -o='python_files=*.py'

You ultimately get a failure:

ImportError while importing test module '/Volumes/gsnedders/projects/wpt/web-platform-tests/webdriver/tests/classic/permissions/set.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'permissions.set'

This is because:

(Pdb) pp sys.modules['permissions']
<module 'permissions' from '/Volumes/gsnedders/projects/wpt/web-platform-tests/webdriver/tests/bidi/external/permissions/__init__.py'>

Per the docs:

This requires test module names to be unique when the test directory tree is not arranged in packages, because the modules will put in sys.modules after importing.

If we instead use the importlib import mechanism, we get:

ImportError while importing test module '/Volumes/gsnedders/projects/wpt/web-platform-tests/webdriver/tests/bidi/browser/create_user_context/__init__.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/bidi/browser/create_user_context/create_user_context.py:3: in <module>
    from .. import get_user_context_ids
E   ImportError: cannot import name 'get_user_context_ids' from 'webdriver.tests.bidi' (/Volumes/gsnedders/projects/wpt/web-platform-tests/webdriver/tests/bidi/__init__.py)

I'm confused as to how it's resolved .. to be webdriver.tests.bidi there (rather than webdriver.tests.bidi.browsers).

[gsnedders]

Note that prepend works if we add __init__.py files all the way up, because we then end up importing unique module names.

[OrKoN]

I tried to do pytest --collect-only recently and the following workaround worked for me:

To webdriver/tests/__init__.py I have added:

import os
from tools.wpt.utils import load_source

here = os.path.dirname(__file__)

localpaths = load_source("localpaths", os.path.abspath(os.path.join(here, os.pardir, os.pardir, "tools", "localpaths.py")))  # type: ignore

and that made the following command work for me:

python3 -m pytest --rootdir=. -c tools/pytest.ini --collect-only -q webdriver/tests/bidi/**/*.py

Normally localpaths are imported in another place that pytest does not know about. This is the output I got https://gist.github.com/OrKoN/e000e32f132db86c102cc35f12fd5852

[gsnedders]

and that made the following command work for me:

python3 -m pytest --rootdir=. -c tools/pytest.ini --collect-only -q webdriver/tests/bidi/**/*.py

Note pytest has significantly different behaviour when arguments (specifying files) are passed, which changes, among other things, how it finds the initial conftest.py.

This is a significance difference to my command in the original issue.