Allow tests to be logged multiple times with pytest-xdist

[davehunt]

Note that this is not related to #927

In pytest-rerunfailures the test protocol is run multiple times in the event of a failure. An enhancement will log each iteration of the test with an outcome of 'rerun' until the maximum number of reruns is reached, at which point the genuine outcome is preserved. This works fine unless running the tests without pytest-xdist, which causes issues when the first rerun of a test is attempted to be removed from the scheduler:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/main.py", line 90, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/main.py", line 121, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/pytest-2.8.3-py2.7.egg/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 521, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 539, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 643, in slave_testreport
INTERNALERROR>     self.sched.remove_item(node, rep.item_index, rep.duration)
INTERNALERROR>   File "/Users/dhunt/.virtualenvs/tmp-aae67d87c3b61cd1/lib/python2.7/site-packages/xdist/dsession.py", line 280, in remove_item
INTERNALERROR>     self.node2pending[node].remove(item_index)
INTERNALERROR> ValueError: list.remove(x): x not in list

For now I can avoid logging multiple reports if we're running tests in parallel, but that does mean that the only indication of multiple tests running will be the multiple logstart calls. Ideally there should be no difference between running tests in serial or parallel.

[nicoddemus]

Hmmm I'm not sure how to implement that other than making xdist explicitly know about "reruns".

    def slave_testreport(self, node, rep):
        if rep.when == "call" or (rep.when == "setup" and not rep.passed):
            self.sched.remove_item(node, rep.item_index, rep.duration)

xdist assumes that a "call" report is enough to remove the item from the scheduler, but that should not be the case when report.outcome == "rerun" as the test will be executed again.

I'm not sure how to solve this without adding new hooks to xdist, for example a hook pytest_xdist_test_finished(node, report) which allows plugins to override the check to know when a slave is done with a test based on the test report.

Another idea would be to grow a new attribute into the report object, something like report.done or report.finished... then xdist can use that instead of rep.when == "call" or (rep.when == "setup" and not rep.passed), and pytest-rerunfailures can set that attribute to False to make it clear the item is not finished yet.

IMHO though I don't see a problem with rerun behaving slightly differently when running under xdist, specially when it comes to reporting only.

[isaulv]

Related: This should affect Box's flaky plugin for pytest.

[RonnyPfannschmidt]

this issue cant be fixed unless we reassert that test node ids are unique

[davehunt]

@nicoddemus & @RonnyPfannschmidt do you have any new thoughts on this? I'm keen to address it, but may need some assistance from one or both of you.

[RonnyPfannschmidt]

this issue is sill unfixable without changing xdist at its core
but we should move it to xdist

[davehunt]

@RonnyPfannschmidt can we move the issue via some admin interface, or would you like me to raise it under xdist and close this one?

[RonnyPfannschmidt]

there is no admin ui to do that, gh is just fancy, not cappable

[davehunt]

Kamino closed and cloned this issue to pytest-dev/pytest-xdist#206