How to retry failing tests? #2905
-
Hi, I have implemented a custom xunit test framework, inheriting from xunit classes and I am executing in parallel. For some reasons (using 3rd party libraries, that do not work well in parallel), there are few tests that fail on the first time they are executed, but if they are run again, they pass. My idea is to retry failing tests for example 3 times. This is the code that I used for that idea:
I have a test method that is always failing on the first run, and succeeding thereafter, but when I run it, I get this summary:
And the output of the test is this:
Can you give me an advice how to do it properly? Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
We have a sample that does this: https://github.com/xunit/samples.xunit/tree/main/RetryFactExample What's happening here is that the message listener in the runner is seeing you run the test multiple times, because you're letting it see the result messages from both the failures and the successes. Our sample shows using a "delayed message bus" so that it hangs onto all the messages while the test runs, and will only ever send those messages along if the test passes, or if the fail count exceeds the limit. |
Beta Was this translation helpful? Give feedback.
-
Hi, it is me again, I was able to do that for Theory as well. The first thing is to change the constructor of the RetryTestCase to pass testMethodArguments as well, like this:
Second, I created my own TheoryDiscoverer in which I am overriding CreateTestCasesForDataRow where I can pass the arguments:
|
Beta Was this translation helpful? Give feedback.
We have a sample that does this: https://github.com/xunit/samples.xunit/tree/main/RetryFactExample
What's happening here is that the message listener in the runner is seeing you run the test multiple times, because you're letting it see the result messages from both the failures and the successes. Our sample shows using a "delayed message bus" so that it hangs onto all the messages while the test runs, and will only ever send those messages along if the test passes, or if the fail count exceeds the limit.