The first iteration of a test is showing as failed, but the retry is showing as passed. The exit code is 0 at the end of the entire run, but the xcresult still surfaces the test as a failure, and that failure cascades up through the test case, test suite, test bundle, and test plan.
Would there not be a way to surface tests as “mixed” or “flaky” if they’ve passed on subsequent retries? Right now there’s no way to distinguish a test that genuinely failed on all attempts from one that recovered on retry. The exit code knows the difference, but the xcresult data doesn’t reflect it.
The first iteration of a test is showing as failed, but the retry is showing as passed. The exit code is 0 at the end of the entire run, but the xcresult still surfaces the test as a failure, and that failure cascades up through the test case, test suite, test bundle, and test plan.
Would there not be a way to surface tests as “mixed” or “flaky” if they’ve passed on subsequent retries? Right now there’s no way to distinguish a test that genuinely failed on all attempts from one that recovered on retry. The exit code knows the difference, but the xcresult data doesn’t reflect it.