pytest cache is empty inspite of that "--lf" option execute all the cases

[c0dehu1k]

Use Case:

1. I have a huge Web automation suite(and yes with flaky test cases).70k test case execution per week.
2. There are some random failures and I want to rerun them and Voila we have a wonderful feature of --lf
3. Now My all test case got passed in third rerun but note I have 4th rerun also with --lf option.
4. 4th Rerun picks all the test case back again. Driving me crazy.

[c0dehu1k]

[RonnyPfannschmidt]

the current version of the --lf flag is intended at developers, with the inception of the feature, ever since the firs versions in pytest-xdist it was considered as a win, to run all tests when no failure was know

this is no longer true for ci usage with bigger testsuites, since there a different focus is desires

a discussion about sound solutions is needed as there is various ways to approach making cache/lastfailed suitable for ci usage

[nicoddemus]

For the OP's exactly problem we might consider adding a new configuration option that makes --lf not run any tests if its failed cache is empty.

This is also useful on the command-line: when I start fixing tests incrementally using --lf, where I fix a few tests and then run pytest --lf -x again to see what tests fail next, I always hit CTRL+C when I see the "no previous failures, running all" message. I would love to have an option which just said all tests fixed, congratulations :) a la tox does. 😁

[nicoddemus]

Which is not to say that we should abandon the idea of how to reuse the failed cache across runs in different CI workers; it is just that IMO those are two separate things.

[brianmaissy]

Should it be a configuration option or the new default behavior? Is it likely that people rely on the current behavior to run all of the tests?

If it should be an option, how's --stop-when-none-failed for a name? (with store_true behavior). Is it accepted to have many-letter aliases like --snf?

in any case happy to implement it

[nicoddemus]

Personally I would like to change the current behavior so it just stops with "success" after no more failed tests are left to run, but we probably want to introduce an option to fallback to the current behavior because I can imagine a scenario where users implemented a way to share the lastfailed cache between CI computers and are always running with --lf all the time; in that scenario they probably want to run all tests when there are no more failed tests on the cache.

[brianmaissy]

ok cool. how should we call the option? --run-all-tests-when-none-failed-or-no-cache seems a bit wordy

[nicoddemus]

I was thinking in terms of an enumeration rather than a bool value, something like lastfailed_no_failures_behavior=stop/runall, but this also seems wordy. Suggestions are welcome. 😛

[nicoddemus]

Fixed in #3295