Support Maven Surefire Plugin rerunning flaky tests #539
Comments
|
Thanks for the detailed description of your use case. There is no option available currently. Rather than changing the logic, I would specifically handle the "flaky" annotations in the test result files: https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html Can you provide an example test result with one failure and success reruns, as well as one result file with all rerun failures? |
|
Sure, for completeness, I made 4 tests fail/error by either throwing an exception or doing a false assertion:
Output of
<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" version="3.0" name="example.ExampleTests" time="0.002" tests="2" errors="1" skipped="0" failures="1">
<testcase name="throwsAlways" classname="example.ExampleTests" time="0.01">
<error message="Always throws exception" type="java.lang.RuntimeException"><![CDATA[java.lang.RuntimeException: Always throws exception
at example.ExampleTests.throwsAlways(ExampleTests.java:26)
]]></error>
<rerunError message="Always throws exception" type="java.lang.RuntimeException">
<stackTrace><![CDATA[java.lang.RuntimeException: Always throws exception
at example.ExampleTests.throwsAlways(ExampleTests.java:26)
]]></stackTrace>
</rerunError>
<rerunError message="Always throws exception" type="java.lang.RuntimeException">
<stackTrace><![CDATA[java.lang.RuntimeException: Always throws exception
at example.ExampleTests.throwsAlways(ExampleTests.java:26)
]]></stackTrace>
</rerunError>
<rerunError message="Always throws exception" type="java.lang.RuntimeException">
<stackTrace><![CDATA[java.lang.RuntimeException: Always throws exception
at example.ExampleTests.throwsAlways(ExampleTests.java:26)
]]></stackTrace>
</rerunError>
</testcase>
<testcase name="failsAssertionAlways" classname="example.ExampleTests" time="0.003">
<failure message="Always fails assertion ==> expected: <1> but was: <2>" type="org.opentest4j.AssertionFailedError"><![CDATA[org.opentest4j.AssertionFailedError: Always fails assertion ==> expected: <1> but was: <2>
at example.ExampleTests.failsAssertionAlways(ExampleTests.java:40)
]]></failure>
<rerunFailure message="Always fails assertion ==> expected: <1> but was: <2>" type="org.opentest4j.AssertionFailedError">
<stackTrace><![CDATA[org.opentest4j.AssertionFailedError: Always fails assertion ==> expected: <1> but was: <2>
at example.ExampleTests.failsAssertionAlways(ExampleTests.java:40)
]]></stackTrace>
</rerunFailure>
<rerunFailure message="Always fails assertion ==> expected: <1> but was: <2>" type="org.opentest4j.AssertionFailedError">
<stackTrace><![CDATA[org.opentest4j.AssertionFailedError: Always fails assertion ==> expected: <1> but was: <2>
at example.ExampleTests.failsAssertionAlways(ExampleTests.java:40)
]]></stackTrace>
</rerunFailure>
<rerunFailure message="Always fails assertion ==> expected: <1> but was: <2>" type="org.opentest4j.AssertionFailedError">
<stackTrace><![CDATA[org.opentest4j.AssertionFailedError: Always fails assertion ==> expected: <1> but was: <2>
at example.ExampleTests.failsAssertionAlways(ExampleTests.java:40)
]]></stackTrace>
</rerunFailure>
</testcase>
<testcase name="failsAssertionOnlyFirstAttempt" classname="example.ExampleTests" time="0.0">
<flakyFailure message="First attempt fails assertion ==> expected: <1> but was: <2>" type="org.opentest4j.AssertionFailedError">
<stackTrace><![CDATA[org.opentest4j.AssertionFailedError: First attempt fails assertion ==> expected: <1> but was: <2>
at example.ExampleTests.failsAssertionOnlyFirstAttempt(ExampleTests.java:32)
]]></stackTrace>
</flakyFailure>
</testcase>
<testcase name="throwsOnlyFirstAttempt" classname="example.ExampleTests" time="0.001">
<flakyError message="First attempt throws exception" type="java.lang.RuntimeException">
<stackTrace><![CDATA[java.lang.RuntimeException: First attempt throws exception
at example.ExampleTests.throwsOnlyFirstAttempt(ExampleTests.java:18)
]]></stackTrace>
</flakyError>
</testcase>
</testsuite>
|
EnricoMi
changed the title
|
Thanks for the input. With the provided XML the action reports one test as error, one as failure and the last two as pass. Can you add another two tests that fail / error twice and pass on third attempt? |
|
Sure, see below for these 7 tests so far:
Output of
|
Draft
|
The action only considers known tags like The |
|
I think you're right and I agree with the current behaviour. After reading your comment, I've dug a bit deeper and found that I had not accurately reproduced the actual problem. Some context: we run ~5000 tests spread over many classes and each class gets its own Though our I'll focus my efforts now on how the xml reports are generated, it might have to be related to the fact we're running our tests in parallel. Thanks for helping along and the quick responses. |
|
Thanks for clarification and for raising this! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
First of all, I think this plugin works great and provides many GitHub reporting options. 👌
I'm currently in the process of running my tests in parallel to speedup my test run. Some of my tests now fail because of concurrency issues and that's why I also configured the Maven Surefire Plugin to rerun failed tests.
However, I notice a few things:
my regular
mvn testcommand ultimately succeeds (i.e. after rerunning some tests, all my tests pass):however,
publish-unit-test-result-actioncreates a check with a failed status (❌Maven Test Results)the annotations show failed tests, since the first attempt they did indeed fail (however, succeeding attempts pass):
Note that my
mvn testcommand completes with the following output (Flakessignifies the reruns):the duration shown in the summary (57m) seems to be the sum of the parallel runs instead of the wall clock time
My simplified workflow:
Log output of the
Publish Test Resultsstep (sensitive parts removed):Example of annotations of 2 different tests where in both cases only the first attempt failed:
Just for completeness, but probably not necessary, relevant parts of
pom.xml:Example of (part of) an xml report file that has a passing test after a rerun:
Is there a configuration parameter I missed that might create a succeeding Check (✅
Maven Test Results) and that causes the report to show only succeeding tests (optionally after retries)? Or is rerunning simply not supported?The repository I'm working on is private, so unfortunately I can't give any direct links, but let me know if more information is needed.
The text was updated successfully, but these errors were encountered: