Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pytest cache is empty inspite of that "--lf" option execute all the cases #3139

Closed
ersuniljoshi opened this Issue Jan 22, 2018 · 9 comments

Comments

Projects
None yet
5 participants
@ersuniljoshi
Copy link

ersuniljoshi commented Jan 22, 2018

Use Case:

  1. I have a huge Web automation suite(and yes with flaky test cases).70k test case execution per week.
  2. There are some random failures and I want to rerun them and Voila we have a wonderful feature of --lf
  3. Now My all test case got passed in third rerun but note I have 4th rerun also with --lf option.
  4. 4th Rerun picks all the test case back again. Driving me crazy.
@ersuniljoshi

This comment has been minimized.

Copy link
Author

ersuniljoshi commented Jan 22, 2018

screen shot 2018-01-22 at 4 00 51 pm

@pytestbot pytestbot added the type: bug label Jan 22, 2018

@ersuniljoshi ersuniljoshi changed the title pytest cache is empty inspite --lf option execute all the cases pytest cache is empty inspite of that "--lf" option execute all the cases Jan 22, 2018

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jan 22, 2018

the current version of the --lf flag is intended at developers, with the inception of the feature, ever since the firs versions in pytest-xdist it was considered as a win, to run all tests when no failure was know

this is no longer true for ci usage with bigger testsuites, since there a different focus is desires

a discussion about sound solutions is needed as there is various ways to approach making cache/lastfailed suitable for ci usage

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jan 22, 2018

For the OP's exactly problem we might consider adding a new configuration option that makes --lf not run any tests if its failed cache is empty.

This is also useful on the command-line: when I start fixing tests incrementally using --lf, where I fix a few tests and then run pytest --lf -x again to see what tests fail next, I always hit CTRL+C when I see the "no previous failures, running all" message. I would love to have an option which just said all tests fixed, congratulations :) a la tox does. 😁

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jan 22, 2018

Which is not to say that we should abandon the idea of how to reuse the failed cache across runs in different CI workers; it is just that IMO those are two separate things.

@brianmaissy

This comment has been minimized.

Copy link
Contributor

brianmaissy commented Jan 25, 2018

Should it be a configuration option or the new default behavior? Is it likely that people rely on the current behavior to run all of the tests?

If it should be an option, how's --stop-when-none-failed for a name? (with store_true behavior). Is it accepted to have many-letter aliases like --snf?

in any case happy to implement it

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jan 27, 2018

Personally I would like to change the current behavior so it just stops with "success" after no more failed tests are left to run, but we probably want to introduce an option to fallback to the current behavior because I can imagine a scenario where users implemented a way to share the lastfailed cache between CI computers and are always running with --lf all the time; in that scenario they probably want to run all tests when there are no more failed tests on the cache.

@brianmaissy

This comment has been minimized.

Copy link
Contributor

brianmaissy commented Jan 27, 2018

ok cool. how should we call the option? --run-all-tests-when-none-failed-or-no-cache seems a bit wordy

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jan 29, 2018

I was thinking in terms of an enumeration rather than a bool value, something like lastfailed_no_failures_behavior=stop/runall, but this also seems wordy. Suggestions are welcome. 😛

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Mar 20, 2018

Fixed in #3295

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.