the behavior of pytest differs a bit bettween mark.usefixutres and usefixtures specified in pytest.ini

[NicolasTriquenaux]

I have acknoldeged a difference in invokation time of fixture teardown is performed when either it is specied with the usefixture marker or throug the usefixture field of pytest.ini

Here is an example :

• test_dummy.py :

import pytest

@pytest.mark.usefixtures("mark_test_outcome")
class Base(object):
    def test_from_base(self):
        assert True


class TestClass4(Base):
    def test_from_class_4(self):
        raise RuntimeError("OOPS")

    def teardown_method(self):
        print "\n> In test teardown method: >> test_failure : {} <<".format(self.test_failure)
        if self.test_failure:
            print "Oh my ! the test have failed ! do specific stuff"
        else:
            print "All is fine, cleaning up"

• conftest.py:

import pytest

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()  # get the report object

    # set a report attribute for each phase of a call, which can
    # be "setup", "call", "teardown"
    setattr(item, "test_" + rep.when, rep)


@pytest.fixture(scope="function")
def mark_test_outcome(request):
    print "\n> In fixture mark_test_outcome setup"
    # Part executed when using the fixture. Here we are only interested in the
    # fixture teardown, so it is empty
    yield
    # We dynamically set a new attribute (test_failure) that can be accessed by the test being
    # under execution with self.test_failure. True means that the test call failed.
    # It only look at that test call not at the setup & teardown invocation.
    print "\n> In fixture mark_test_outcome teardown : {}".format(request.instance)
    request.instance.test_failure = request.node.test_call.failed

• pytest.ini :

Here is the outcome :

test_dummy.py::TestClass4::test_from_base 
> In fixture mark_test_outcome setup
PASSED
> In fixture mark_test_outcome teardown : <test_dummy.TestClass4 object at 0x7f485eeb4250>

> In test teardown method: >> test_failure : False <<
All is fine, cleaning up

test_dummy.py::TestClass4::test_from_class_4 
> In fixture mark_test_outcome setup
FAILED
> In fixture mark_test_outcome teardown : <test_dummy.TestClass4 object at 0x7f485eeb4490>

> In test teardown method: >> test_failure : True <<
Oh my ! the test have failed ! do specific stuff

I do see that the teardown of the fixture mark_test_outcome is called at the begining of the teardown_method which is what i do look for.

However, if instead of decorating the test class with @pytest.mark.usefixtures("mark_test_outcome") i use the field usefixture of the pytest.ini file the behavior is different. Here is the diff with the setup above and its oucome:

• test_dummy.py:

- @pytest.mark.usefixtures("mark_test_outcome")
class Base(object):

• pytest.ini:

+ [pytest]
+ usefixtures = mark_test_outcome

Outcome:

_________________________ ERROR at teardown of TestClass4.test_from_base _________________________________

self = <test_dummy.TestClass4 object at 0x7f29954715d0>

    def teardown_method(self):
>       print "\n> In test teardown method: >> test_failure : {} <<".format(self.test_failure)
E       AttributeError: 'TestClass4' object has no attribute 'test_failure'

test_dummy.py:14: AttributeError
________________________ ERROR at teardown of TestClass4.test_from_class_4 _________________________________

self = <test_dummy.TestClass4 object at 0x7f2995478550>

    def teardown_method(self):
>       print "\n> In test teardown method: >> test_failure : {} <<".format(self.test_failure)
E       AttributeError: 'TestClass4' object has no attribute 'test_failure'

test_dummy.py:14: AttributeError

So here the invokation of the mark_test_outcome teardown is performed later (i assume at the end of the teardown_method in that case) which render my attribute injection erroneous.

So down to my question : is this expected to have such a difference ? or is it a bug and both ways should have the same behavior. I am using pytest 4.6.11. I know it is an old version, is a newer version fix that problem ?

[RonnyPfannschmidt]

please try with a recent version as well

[RonnyPfannschmidt]

that looks like a bug, some investigation is needed to review how exactly we interpret usefixtures from conf vs marks

[RonnyPfannschmidt]

according to

pytest/src/_pytest/fixtures.py

Lines 1412 to 1419 in 1003bea

	The FuncFixtureInfo object holds information about fixtures and FixtureDefs
	relevant for a particular function. An initial list of fixtures is
	assembled like this:
	- ini-defined usefixtures
	- autouse-marked fixtures along the collection chain up from the function
	- usefixtures markers at module/class/function level
	- test function funcargs

it seems like it is intended

based on how we collect i suspect a fixup wrt autouse fixure ordering may be involved as well
some history digging is needed later (im pretty sure i wont get to it before next week, it may slip)

[NicolasTriquenaux]

That answers the question. Should i open an issue linking that conversation ? and mark the conversation as revolved ?

[RonnyPfannschmidt]

Sounds like a plan, please do

[NicolasTriquenaux]

@RonnyPfannschmidt : i oppended the issue, sorry for the delay