Supporting flaky tests - just thinking out loud here #9839
Unanswered
jeffwright13
asked this question in
Ideas
Replies: 1 comment 1 reply
-
Definitely something a plugin could do, via the |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am aware of the Pytest plugins for flakiness (pytest-flaky, and pytest-rerun). Those are great if I insist on a rigid determination of P or F for each test for a given run. But what would also be valuable is the ability to actually mark a test as Flaky (or maybe Indeterminate). Once marked as such, the test's outcome would occupy a new category, Flaky or Indeterminate or whatever it needs to be called. Tests marked Flaky would not count toward P or F or anything else; they would instead occupy their own category. And maybe they could be notated on the command line as a snowflake:
.F..❅.s...F....sXxFF....❅..
A good tester using this feature would always strive to improve their tests and systems-under-test so that the number of Flaky category tests is zero...but knowing that in the real world, having a zero-flake test run is something of a unicorn. :-)
I am guessing this would require work in the guts of Pytest, and not be available for someone to develop as a plugin, but I'm not sure.
What do you think? Is this a terrible idea? Would it be something the devs could do, or is it something a plugin might be able to implement?
Beta Was this translation helpful? Give feedback.
All reactions