Skipping in a teardown part of a yield fixture is not counted properly

[timhoffm]

I would like to delay skipping tests to the end of the test execution. To do so, I yield from a fixture and call pytest.skip afterwards. This is part of a more complex fixture, but boils down to a minimal example of:

@pytest.fixture()
def post_exec():
    params = {'skip': False}
    yield params
    if params['skip']:
        pytest.skip('This is skipped')

def test_skip(post_exec):
    post_exec['skip'] = True
    # do further stuff

Expected result: The test is skipped.

Actual result: The test appears twice in the statistics

pytest tests/test_x.py::test_skip --v

[...]

tests/test_x.py::test_skip PASSED                                                                  [100%]
tests/test_x.py::test_skip SKIPPED (This is skipped)                                               [100%]

================================================ short test summary info =================================================
SKIPPED [1] tests/text_x.py:207: This is skipped
============================================== 1 passed, 1 skipped in 0.02s ==============================================

Tested with pytest 6.4.2.

[bluetech]

Pytest executes tests in three phases - setup, run, teardown. Each phase is reported separately as soon as it completes. Normally setup and teardown are invisible in the reporting when they pass. But if they fail or are skipped, then they are reported separately. So the PASSED here is from the run phase, and the SKIPPED is from the teardown phase.

I think that conceptually this is good. For example, there is a difference between a test failing in its run phase and failing in its teardown phase. What I think could be improved is to somehow indicate that the report is for setup/teardown, at least in -v output.

As for making skip during teardown show up as a skip of the entire test (also the run phase), I don't think it's possible given the current pytest architecture, and I personally don't really think it's desirable either. Perhaps though you could describe more what makes you want to skip during teardown, it is unusual.

[timhoffm]

Thanks for the explanation!

The distinction between phases certainly makes sense for FAILED. Not quite so sure about SKIPPED but I see that it would be hard to handle that differently. I agree that the reporting should be improved. The simplest improvement would be to add "in teardown" to the long format like this:

tests/test_x.py::test_skip PASSED                                                                  [100%]
tests/test_x.py::test_skip SKIPPED in teardown (This is skipped)                                   [100%]

Docs

https://docs.pytest.org/en/latest/reference/reference.html#pytest-skip says

This function should be called only during testing (setup, call or teardown) [...]

This made me assume skip in call and teardown are equivalent. It would be good to indicate here that there is a difference when calling in the different phases.

Why I want to skip at the end

I'm writing a plugin for regession testing. The usage is essentially:

def test_x(regression):
    x = calculate_x()
    regression.check_data(x)

Where regression.check_data() can obtain stored reference data for x and compares with the passed value. To be able to easily create these reference data, the regression can run in different modes. In mode 'create', it will take the passed value and save it. In mode 'amend' it will use existing data for comparison, but if a new test was added that now does not have reference data, again the passed value will be saved. In the case of saving new data I want to indicate that the test did not actually test content, which I do by skipping it.
In the simple case, I can do that within check_data(), and that works. However, data can be complicated, which may make it necessary sometimes to have multiple checks with assertion test functions (similar to the need of sometimes having multiple asserts in a test function). Therefore, a more realistic use case could look like

def assert_ytype_equal(actual, desired):
    ...

def test_xy(regression):
    x, y = calculate_xy()
    regression.check_data(x, comparator=nptest.assert_allclose)
    regression.check_data(y, comparator=assert_ytype_equal)

In that case, I need both check_data functions to execute when I want to save data. I therefore cannot skip from within the check_data() function. Thus my idea was to just set a flag that data was created regression.check_data() and defer the skip call to the teardown part of the regression fixture.

I appreciate suggestions how to best handle such a situation. - Well technically it works. Only the reported pass/skip values are surprising.