Junit & HTML reports - Incorrect count for failed tests when -retry-tests-on-failure flag is passed

[ssharma0312]

With introduction of XCode 13, we can now re-run failed tests multiple times. This feature is very helpful for flaky tests.

Unlike .xcresult file, the test reports produced by XCTestHTMLReport counts re-tried tests x number of times (where x == no. of times a test is retried). This shows incorrect metrics.

In the following example I am passing flag -retry-tests-on-failure, so my failed test will re-run until they pass but for maxi 2 times (-test-iterations )
xcodebuild -workspace ABC.xcworkspace -scheme ABCTest -sdk iphonesimulator -destination 'platform=iOS Simulator,name=iPhone 11,OS=15.0' test -parallel-testing-enabled YES -parallel-testing-worker-count 5 -maximum-concurrent-test-simulator-destinations 5 -test-iterations 2 -retry-tests-on-failure -quiet -resultBundlePath TestResults

The .xcresult file produced, the "Failed" tab only shows 3 failed test because these 3 failed on both retries, the few tests that were failed on 1st attempt but succeed on 2nd retry are marked as Passed and counted only once, and showed under "Passed" tab. XCTestHTMLReport plugin should follow same pattern i.e. the tests that were passed on retry should not be counted as failed in HTML & Junit reports & should be counted only once.

.xcresult file

.HTML Report produced by XCTestHTMLReport

.HTML Report produced by XCTestHTMLReport

[nacho4d]

@ssharma0312 Thank you for reporting this.
I haven't reproduced this bug but since you provided all the screenshots and information I think this look like a valid bug (or a new feature in Xcode13 XCTestHTMLReport doesn't support yet and it looks like a bug).

[erik-agens]

We are also seeing this. An awesome improvement in Xcode 13.
Maybe follow .xcresult's pattern and show Mixed (xx) for tests that have failed, but succeeded on retry?

[ssharma0312]

Any update on this bug?

[Husseinovich]

I am also seeing this issue on Xcode 13.1. Tests that passed on successive iterations are reported as failure. Very critical bug.

[ssharma0312]

Working on this bug, I will create a MR coming shortly with the fix.

[tylervick]

Please feel free to test by pulling from #247

Or using mint: mint run tylervick/XCTestHTMLReport@tvick/mixed-test-status, though I don't encourage the practice of executing arbitrary git repos (even from myself 😬)..