Report mixed tests as success in junit if the last iteration passed

[acecilia]

👋

This PR introduces the system-err XML tag for the junit report. You can see how this is used by the teamcity junit parser here.

This way, now when there are multiple iterations of a test, the final result of the test is the status of its last iteration. For example, having two iterations:

• If the first succeeds, test is shown as passed
• If the first fails and the second succeeds, test is shown as passed
• If both fail, test is shown as failed

Failed iterations that are not the last one are reported using the system-err tag.

I would say this makes more sense than previous implementation, as usually when you have multiple iterations of a test is to fight against flakiness.

See for example how this looks after teamcity parses the junit XML:
In the test history:

In the test log:

I believe that the main issue here is that junit reports do not support reporting multiple iterations of a test 😢

[acecilia]

@tylervick Hey 👋 🙂 Any chance of getting your opinion on this? 🙏

[tylervick]

Thanks for this! It's frustrating that Junit doesn't support the "iterations" concept.

My only question - Is there a discernible difference between relying on the last iteration versus any successful iteration period?

For instance, what is the expected outcome when a fixed number of iterations emits mixed results?

[acecilia]

Thanks so much for taking the time to review this 🙌

Thanks for this! It's frustrating that Junit doesn't support the "iterations" concept.

Yes 100%. I was looking for a different popular test report format that could support it, but surprisingly I did not find a good candidate. Most iOS tooling relies on junit. Let me know if you know about any alternative :)

Is there a discernible difference between relying on the last iteration versus any successful iteration period?

I would say so. It depends on the expectation a developer has when repeating the execution of a test:

1. To avoid flakiness: I believe the most common case when running a test more than once, is to avoid flakiness. Example: running UITests, test repetition is a must due to the inherent flakiness of such tests. Under this scenario the most representative outcome of multiple repetitions is the result of the last repetition. Xcodebuild allows this with the flag -retry-tests-on-failure
2. To detect flakiness: other common usecase is to run a test multiple times with the expectation of detecting if it is flaky. In this case each test is run X times, or until it fails. Under this scenario the most representative outcome of multiple repetitions is the result of the last repetition. This is supported by xcodebuild with the flag -run-tests-until-failure.

For instance, what is the expected outcome when a fixed number of iterations emits mixed results?

This scenario, which is running each test X times despite the outcome, seems to me not very practical and I am not clear on what usecase it has:

• If the expectation is to show a failed test result if all iterations fail, the option 1 above is more performant
• If the expectation is to show a failed result if any of the iterations fail, the option 2 above is more performant
• If the expectation is to show a passed test result if all iterations pass, the option 2 above is more performant
• If the expectation is to show a passed test result if any of the iterations pass, the option 1 above is more performant

What do you think, are you able to identify any other usecase I am missing?

The way I see it, if we were to make a decision on what iteration is the one that represents better the result of a test with mixed iterations, based on the usecases above I would say is the last iteration

[tylervick]

Thank you for the detailed explanation! Completely agree on the two primary use cases for iterations.

FWIW passing -test-iterations without -retry-tests-on-failure or -run-tests-until-failure is what Xcode calls a "fixed number of iterations" in their UI. I agree that it doesn't really make sense here, and especially in the context of a junit report.