Repeat on failure #53 #214
Conversation
| { | ||
| TYPE, METHOD | ||
| }) | ||
| public @interface RepeatOnFailure |
| @@ -0,0 +1,24 @@ | |||
| package com.xceptance.neodymium.module.statement.repeat; | |||
|
|
|||
| public class RepeatOnFailureData | |||
|
|
||
| import com.xceptance.neodymium.module.StatementBuilder; | ||
|
|
||
| public class RepeatStatement extends StatementBuilder |
| public static AtomicInteger val = new AtomicInteger(0); | ||
|
|
||
| @Test | ||
| public void testVisitingHomepage() |
There was a problem hiding this comment.
We might want to have some more meaningful test case names here.
| import com.xceptance.neodymium.module.statement.repeat.RepeatOnFailure; | ||
|
|
||
| @RunWith(NeodymiumRunner.class) | ||
| public class OverridingRepeatOnFailureTest |
There was a problem hiding this comment.
This test is not testing any override.
| } | ||
| catch (Throwable t) | ||
| { | ||
| if (iterationIndex != maxRepetitionNumber) |
There was a problem hiding this comment.
When we interact with a browser, we will need a fresh browser instance here. Otherwise we will reuse the same browser with cookies and session data from the previously failed test.
| }) | ||
| public @interface RepeatOnFailure | ||
| { | ||
| int value() default 1; |
There was a problem hiding this comment.
The value currently represents the number of maximum runs. So if we have it set to 1, we'll run only once even if the test fails. The semantic of @RepeatOnFailure(1) is that the test is repeated once on fail therefore has two runs.
We should change the meaning of the value so that @RepeatOnFailure(1) has one repeated run (or in other words two runs).
| public String getTestName(Object data) | ||
| { | ||
| RepeatOnFailureData d = (RepeatOnFailureData) data; | ||
| return "Run number " + d.getIterationNumber(); |
There was a problem hiding this comment.
"Run number X" is not super obvious as a retried test case run to see in a report. Maybe we could do something more eyecatching.
Maybe keep the original run with the original name, and add an uppercase "RETRY X" or something to it.
There was a problem hiding this comment.
We should investigate if we can improve the current implementation in terms of reporting.
In the jUnit/Allure report, we a lot of skipped runs if the run succeeded before the maximum was reached. Also we always see all failed (repeated) runs, will be go as errors into any report statistic. It would be cool if we could configure to suppress such failed cases which went green on retry (since the test case is known to be flaky otherwise we would not add this annotation).
There was a problem hiding this comment.
Please investigate if removing the "skipped" and "failed" runs from the reporting can be done easily, otherwise we can add an improvement in a separate task.
| } | ||
|
|
||
| @Override | ||
| public String getCategoryName(Object data) |
There was a problem hiding this comment.
This leads to a separate category in the junit test report for all tests using this annotation. It would be nicer if only test reruns land in a separate category and tests which ran successfully just go into a default category.
No description provided.