Option to retry N times until test passes

[vt-alt]

What's the problem this feature will solve?

Some tests are intermittently fail, For example distribution could run tests for multiple packages on quite loaded server (because things are parallelized)) and in that case race conditions occur. Or test could be coded unreliably. There could be slow virtual machine that cause timeouts. These factors could change environment to be sometimes different from when upstream run tests manually on their desktop. If manual run fails upstream developer could just rerun it again. But when we run multiple tests in automated way this could break workflows, send alerts, etc. making life of package maintainers harder.

Describe the solution you'd like

ctest have option like this

  --repeat until-pass:<n>      = Allow each test to run up to <n> times in order to pass

This could be good for pytest to support something similar to help with 'probabilistic' tests.

Fixing test could be very time consuming task, end especially for subtle race conditions. In that case package maintainers tend to disable tests sometimes skipping whole test run. Of course this is wrong strategy but this happens.

As alternative to this maintainer could run pytest with something like --repeat-until-pass 3 greatly lowering probability of false positive test failures.

Alternative Solutions

Alternative solutions are: to fix test (hard and human time consuming); to not run particular test (sometimes this covers almost all tests); to not run test suite at all (stains karma); to repeat whole test run (could take very long time when there are a lot of tests).

[The-Compiler]

The pytest-repeat plugin does exactly this.

[vt-alt]

@The-Compiler No it does different thing or even contrary to my request. Can you reconsider? 🙏

pytest-repeat repeats fixed number of times or if combined with -x it repeats until a failure.

I requested to repeat until success. For example if test is succeeded on the first run it should not be repeated.

Repeating after success increases probability of test failure and this is contrary to the goal that I described. Goal of my request is to decrease probability of (intermittent) test failure by repeating after failure.

[The-Compiler]

Whoops, my bad - I was thinking of pytest-rerunfailures, not pytest-repeat. Sorry!

[vt-alt]

pytest-rerunfailures,

Thanks much! (I already managed to fill feature request on pytest-repeat.)