pytest_addoption is called after CLI argparsing when conftest is in a subdirectory of tests and the path is not specified

[dcermak]

I have hit a rather confusing edge case when using pytest_addoption in a conftest.py that resides in a sub-directory of the tests directory. It appears that pytest is then performing the CLI parsing before collecting the tests and the conftest.py (which leads to really weird issues). On the other hand, when I specify the test path manually, then pytest will do "the right thing" (= perform the CLI parsing after readin in conftest from the directory).

This can be reproduced as follows:

$ poetry init --name "test" --python "^3.9" --description "Just a test" --author "me" -n --dev-dependency="pytest"
$ poetry install
$ mkdir -p tests/base/
$ touch tests/__init__.py
$ cat <<EOF > tests/base/conftest.py
def pytest_addoption(parser):
    parser.addoption("--foobar")
EOF
$ cat <<EOF > tests/base/test_something.py
def pytest_addoption(parser):
    parser.addoption("--foobar")
EOF

Now, I would intuitively expect that I could now run poetry run pytest --foobar but this does not work:

$ poetry run pytest --foobar 
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --foobar
  inifile: None
  rootdir: /tmp/pytest-test

However, if I explicitly set the path where to look for tests, then it works:

$ poetry run pytest --foobar tests/base/*py
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /tmp/pytest-test
plugins: cov-3.0.0
collected 1 item                                                                                                                                                                                                                            

tests/base/test_something.py .                                                                                                                                                                                                        [100%]

============================================================================================================= 1 passed in 0.01s =============================================================================================================

Please also note that pytest is still executing conftest.py if I don't specify a CLI flag. One can for example modify pytest_addoption(parser) as follows:

def pytest_addoption(parser):
    raise ValueError("foo")

And executing poetry run pytest leads to the expected error:

$ poetry run pytest 
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /tmp/pytest-test
plugins: cov-3.0.0
collected 0 items / 1 error                                                                                                                                                                                                                 

================================================================================================================== ERRORS ===================================================================================================================
_______________________________________________________________________________________________________ ERROR collecting test session _______________________________________________________________________________________________________
/home/dan/.cache/pypoetry/virtualenvs/test-HwDUSR58-py3.9/lib/python3.9/site-packages/pluggy/_manager.py:115: in register
    hook._maybe_apply_history(hookimpl)
/home/dan/.cache/pypoetry/virtualenvs/test-HwDUSR58-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py:300: in _maybe_apply_history
    res = self._hookexec(self.name, [method], kwargs, False)
/home/dan/.cache/pypoetry/virtualenvs/test-HwDUSR58-py3.9/lib/python3.9/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
tests/base/conftest.py:3: in pytest_addoption
    raise ValueError("foo")
E   ValueError: foo
========================================================================================================== short test summary info ==========================================================================================================
ERROR  - ValueError: foo
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================================= 1 error in 0.07s ==============================================================================================================