fixture assert is getting overwritten in pytest_collection_modifyitems

[jay746]

Please create 2 files in any folder with below codes:

1. test_py_issue.py

import pytest

def idfn(fixture_value):
    return 0

def create_test():
    import somerandomPACKAGE # inserting error in fixture
    return ["a", "b", "c"]

@pytest.fixture(scope="function", params=create_test(), ids=idfn)
def op_test_config(request):
    return request.param

def test_py_collect_issue(op_test_config):
    print(f"\nHello I am a debug test! and config: {op_test_config}")

2. conftest.py

def pytest_collection_modifyitems(config, items):
    if len(items) !=3:
        print(len(items))
        assert 0, "expected 3 tests to be created!" # commenting this line will give actual error

Here the output is

INTERNALERROR>     assert 0, "expected 3 tests to be created!" # commenting this line will give actual error
INTERNALERROR> AssertionError: expected 3 tests to be created!
INTERNALERROR> assert 0`
But the issue raised in fixture , which is import issue. so pytest should print first raised issue. 
`Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_py_issue.py:11: in <module>
    ???
test_py_issue.py:7: in create_test
    import somerandomPACKAGE # inserting error in fixture
E   ModuleNotFoundError: No module named 'somerandomPACKAGE'`

pip list:

Pillow                  9.2.0              
pip                     20.0.2             
platformdirs            2.5.2              
pluggy                  0.13.1             
psutil                  5.4.2              
py                      1.11.0                         
pyasn1                  0.4.8              
pycodestyle             2.9.1              
pyflakes                2.5.0              
pylint                  1.9.3              
pymongo                 3.7.2              
pyparsing               3.0.9              
pytest                  6.2.2              
pytest-forked           1.3.0              
pytest-html             3.1.1              
pytest-logger           0.5.1              
pytest-metadata         2.0.2              
pytest-random-order     1.0.4              
pytest-repeat           0.9.1              
pytest-timeout          2.0.2              
pytest-xdist            2.4.0              
python-dateutil         2.8.2

Downloads.zip

[JosiasAurel]

Hey, I just tested your code and it works for me.
Here is what I have
In test_py_issue.py

import pytest

def idfn(fixture_value):
    return 0

def create_test():
    import numpy as np # inserting error in fixture
    return ["a", "b", "c"]

@pytest.fixture(scope="function", params=create_test(), ids=idfn)
def op_test_config(request):
    return request.param

def test_py_collect_issue(op_test_config):
    print(f"\nHello I am a debug test! and config: {op_test_config}")

In conftest.py

def pytest_collection_modifyitems(config, items): 
    if len(items) !=3: 
        print(len(items)) 
        assert 0, "expected 3 tests to be created!" # commenting this line will give actual error

[jay746]

Hi @JosiasAurel,
Thanks for replying back on issue.
I see that in fn create_test(), you have replaced the line of code which was introducing the import error.
If you don't insert in the error in fixture then ofcourse it will work fine

As I have mentioned in description, if we have assert/exception raises in fixture , then that is getting replaced by assert of pytest modification block. Which doesn't represent the correct source of issue.

Please add any exception/assert in fixture function and try.

[jay746]

So the expectation that the fixture assert should be printed on console, but pytest hijack exception raises in fixture and prints assert of pytest_collection_modifyitems.

Which looks fishy

[RonnyPfannschmidt]

This works as intended,

the failure in the parameter generator triggers a collection error which is expected and reported

This causes the assertion in the modifyitems hook to go haywire (assertions there are indeed internal errors and not supposed to be used there )

The error could perhaps be enhanced a little

[jay746]

There is usecase that I have to keep the assert in collection modification block to validate the originality of test. But overriding of exception causing quite challenging debug.
Can pytest atleast prints the whole error? The error which comes in fixture + collection modification, which is right way to show and followed in general in python world.
(If one exception causes another then both are showing and traces as well)

[RonnyPfannschmidt]

The collection modifyitems hook runs after the error was handled

[jay746]

Does handing error means it's get printed on console?
I don't see collection error getting printed on console

[RonnyPfannschmidt]

Your assertion in a place where that Type of error should happen exits pytest before the normal printing

[jay746]

Sorry to reiterate the problem, as I see pytest work in order 1) collection ->2) collection modification
Now in above example which I have given, the import error in fixture which is getting called at a test collection so pytest should print the collection error. Not the error which comes after it.
I mean how come we print the last exception not the original one?

[RonnyPfannschmidt]

The error is stored as a report to be printed in the summary, your assertion in the place where you should not asset breaks pytest before it gets to the printing