pytest_runtest_protocol hook not hit for tests marked with pytest.mark.flaky

[eleanorjboyd]

Hi! I am working on a pytest plugin and I am getting an unexpected behavior. For the following test I cannot get the hook pytest_runtest_protocol to work. I try to just have it print when it reaches that hook but it does not show up. I run another test, without the flaky marker and it dos hit the pytest_runtest_protocol. I even try them in the same file and the same behavior exists. Is pytest_runtest_protocol supposed to be skipped for tests marked with flaky?

test that isn't working with the hook

import pytest


@pytest.mark.flaky(reruns=0)
def test_flakyreruns():
    assert False

test that is working with the hook

def test_abc():
    assert False

(these tests are from the same file)

Thanks!

[RonnyPfannschmidt]

Due to lack of information, there's nothing actionable

The hook is called on all items

[nicoddemus]

@eleanorjboyd you mean the flaky marker from https://github.com/box/flaky?

That plugin overwrites the pytest_runtest_protocol:

https://github.com/box/flaky/blob/69c297e516846257dda83d3b57f6618dc6f1d4dc/flaky/flaky_pytest_plugin.py#L51

Depending on how your own plugin is getting installed, flaky might be getting called instead of yours, given pytest_runtest_protocol is meant to be unique.

You might try to use a hookwrapper instead.

[eleanorjboyd]

Hello! Sorry for the lack of information. The additional library was actually pytest-rerunfailures. Great suggestion on changing to hookwrapper to avoid being overwritten, that was the fix to my issue! Where is it noted which hooks are unique? Im sure it says somewhere in the API doc but am having trouble finding it. Thanks

[nicoddemus]

It says so in the bottom of the hook docs:

Stops at first non-None result, see firstresult: stop at first non-None result. The return value is not used, but only stops further processing.

It might not be as explicit as possible, but it means that it will only call hooks until one of them returns non-None, meaning only one hook can actually execute the test protocol.

[eleanorjboyd]

Aha yes that makes sense, great thanks!