Rerun failed tests automatically and detect flaky tests

[pihme]

Description

• Use settings in surefire to rerun failing tests. http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
• Scan the log for flaky tests
• Fail build when flaky tests were found
• collect names of flaky tests and display them prominently

[pihme]

Issue is postponed until prioritization of flaky tests is discussed.

[pihme]

@menski I could use your help here. I googled my way to a PoC, but I struggle to get it running on the Zeebe Pipeline.

The PoC is here: https://github.com/pihme/jenkinsTestBed/tree/rerun-failed-tests

It works as follows:

• runTest.sh runs the tests and then scans the log for flaky tests; if it finds them it collects the interesting lines, extracts the test names and writes them into a file, and fails the stage of the build
• Jenkinsfile then looks for the file, and if it's there writes the content into the build description

In Zeebe the second step seems to be not working. Maybe it's because of the "container" tag and the file I write isn't visible to the world outside.

Besides that there is potential for improvement:

• in the PoC I didn't capture the exit code of the original maven run. This would be important to mark the stage as failed if it's an ordinary test failure
• I do the extraction of the tests in two steps: first I use grep to look for the lines I am interested in, and then I use awk to extract a certain "column". I guess both steps could be combined into an elegant regex with capture groups
• now that I think of it, the code will also report tests that always fail as flaky. It needs a better matcher:

[WARNING] Flakes: 
[WARNING] com.github.pihme.jenkinstestbed.module1.FlakyTest.shouldFailOnceInTwoInvocations
[ERROR]   Run 1: FlakyTest.shouldFailOnceInTwoInvocations:22 failed
[ERROR]   Run 2: FlakyTest.shouldFailOnceInTwoInvocations:22 failed
[INFO]   Run 3: PASS

vs a test that always fails:

[ERROR] com.github.pihme.jenkinstestbed.module1.FlakyTest.shouldFailNineInTenInvocations
[ERROR]   Run 1: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 2: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 3: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 4: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 5: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 6: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 7: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 8: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 9: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 10: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 11: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 12: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 13: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 14: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 15: FlakyTest.shouldFailNineInTenInvocations:15 failed
[ERROR]   Run 16: FlakyTest.shouldFailNineInTenInvocations:15 failed

I will continue working on this myself.
But I will also appreciate some help here.

[menski]

@pihme is there a branch where I can see this applied to the Zeebe Pipeline, as you mentioned that the second step is failing there?

[pihme]

https://github.com/zeebe-io/zeebe/tree/4347-detect-flaky-tests

[pihme]

@menski: Made several improvements. It is now so close to something usable that the missing piece is all the more infuriating.

The missing piece is the capture of the exit code of a piped command. In my toy project this works with:

mvn test -Dsurefire.rerunFailingTestsCount=15 | tee test.txt

STATUS=${PIPESTATUS[0]}

In Zeebe it doesn't. Might have to do with the first line in the script. In my toy project I use #!/bin/bash, in Zeebe it is just #!/bin/sh which may or may not be a bash shell. (oh, how I love Linux)

Any help is welcome.

[menski]

@pihme regarding the PIPESTATUS this is a bash feature, and you are correct you have to run it as #!/bin/bash script to use it. But that is okay. We normally try to use sh as minimum in case we have an image which does not expose bash at all. But for the maven commands it's fine to switch to bash. Let me know if you need help here.