Native Support for Xcode Test Coverage (XCResult) format.

[james-rant]

Is your feature request related to a problem? Please describe.
The latest uploader converts the Xcode test coverage file formats into another format before uploading. This takes a lot of time (sometimes up to 10 minutes realtime (which is multiplied for running on a Mac), on our CircleCI workflows). This introduces a lot of wasted time waiting for the conversion before uploading, and costs us money as we pay per minute. We can't parallelise this job, because it requires the building and testing to complete before uploading the coverage. our project is quite small (around 5k lines of Swift), but will eventually become much larger and I'm worried that this time to convert will increase linearly as we add more tests.

Describe the solution you'd like
The uploader should support Xcode's native test coverage format to remove the time it takes to convert the files. Failing that, the uploader should upload the raw test coverage files and do the conversion and processing on codecov's backend.

Describe alternatives you've considered
We're exploring Xcode Cloud, and may consider moving away from CircleCI - of course this means we would no longer use CodeCov as Xcode Cloud supports its own test coverage format.

Additional context

[drazisil-codecov]

Copied to Canny as a feature request: https://codecov.canny.io/feedback/p/native-support-for-xcode-test-coverage-xcresult-format
@codecov/product

[bobergj]

Improving the performance of the conversion step is a blocker for us, not an "enhancement" (as currently tagged). It takes over 10 minutes to process our xcresult bundle on a CircleCI runner. Adding codecov would thus increase our test jobs runtime by 50%. We were considering starting a paid plan and give us your money, but as is we can't.

Happy to provide you with a sample xcresult (through a private channel).

[jeffremer]

Hopefully this isn't just viewed as piling on, but given that it's been six months since there's any update, I'm hoping there's some concrete on the roadmap for this. The canny.io link from @drazisil-codecov above doesn't appear to work - I'm presented with a sign-in prompt even when signed up.

We're evaluating Codecov and have it set up to upload xcresult bundles with the --xcode --xcodeArchivePath [XCRESULT] options, which appears to work, but it takes a long time to complete the Running xcode coversion... [sic] part of things.

As potential customers, what can we expect here? I'm a little disappointed and surprised by what I've encountered so far:

• It's non-obvious that --xcode is required when --xcodeArchivePath is specified - maybe I'm missing something, but the uploader doesn't do anything expected when you just try passing --xcodeArchivePath without the --xcode flag. Instead the uploader finds an unrelated fastlane report and then complains about it being invalid. Shouldn't the presence of --xcodeArchivePath alone be enough to make the uploader try to use that?
• A typo in the Running xcode coversion log statement - it makes me wonder if this feature is well tested and supported. I get it, typos happen, but that particular line of code has been in the uploader for nearly a year and not fixed - not confidence inspiring.
• It takes as long to convert and upload the coverage as it does to build and run the entire test suite in CI.

I'm really not trying to just complain here, but as is, Codecov doesn't feel useable for us and it's honestly pretty surprising since I have successfully used Codecov in the past. The conversion and upload time is a major blocker. We could somewhat modularize the jobs and have the coverage conversion and upload kicked off later in the pipeline so developers can get their test results faster, then see coverage sometime after that, but this doesn't feel like something we should have to do - ideally the Codecov upload would be fairly quick, as quick as I've seen it in the past.

Given the lack of traction (unless I'm missing something) I'm inclined to say we will probably have to rule out Codecov, but I'd love to be proven wrong here. Do you have customers using the new uploader on non-trivial iOS apps?

[james-rant]

We've since moved away from Codecov and CircleCI. Our project increased in complexity such that the Codecov conversion step was taking almost double the time it took to build and run our test suite, so half the cost we were paying for CircleCI was for Codecov's slow uploader alone. Completely unacceptable. We're now using Xcode Cloud and Xcode's native test coverage reports, as Codecov seems entirely uninteresting in solving this problem.

[thomasrockhu-codecov]

@james-rantmedia @jeffremer @bobergj, I'm not an xcode developer, but I'm interested in taking a crack at trying to work on this. Do you have an open-source project that is running into this issue?

[jeffremer]

@james-rantmedia @jeffremer @bobergj, I'm not an xcode developer, but I'm interested in taking a crack at trying to work on this. Do you have an open-source project that is running into this issue?

Thanks for the reply, but no @thomasrockhu-codecov - our project is proprietary.

[thomasrockhu-codecov]

@jeffremer, thanks for getting back so quickly. Would you be able to share how you are collecting coverage?

[jeffremer]

We have code coverage enabled in a XCTest plan for all the pertinent targets and the coverage information ends up being stored in the resulting xcresult bundle. This is viewable directly in Xcode, but I've also used third party tools such as xcov to generate HTML reports. xcov only takes a few seconds to generate a coverage report from an xcresult bundle.

[thomasrockhu-codecov]

@jeffremer, I made some changes to the uploader (v0.4.0). Can you try adding --xs to the codecov command (e.g. ./codecov --xs

[thomasrockhu-codecov]

@bobergj this might be helpful for you too. Any numbers about how long it takes would be helpful too

[jeffremer]

@thomasrockhu-codecov that's a big improvement. Results from time ./codecov -t [REDACTED] --xs fastlane/test_output/[REDACTED].xcresult

[thomasrockhu-codecov]

@jeffremer woo hoo! glad that it helped!