Add Support for Xcode 11 xcresult & xccovreport

[abotkin-cpi]

With Xcode 11, Apple has changed up the file format of xcresult so that you can't simply inspect into the xcresult's file structure. Instead, you need to go through their tool xcresulttool in order to retrieve files like xccoverage out of the xcresult.

[abotkin-cpi]

For anyone who comes upon this, our intern worked on creating a Swift command line tool (http://github.com/ChargePoint/xcparse) that can help in extracting files like the code coverage report out of Xcode 11+ xcresult files. It's installable via Homebrew & we're using it in our cloud CI to workaround this. Once you extract out the action.xccovreport using xcparse, your Dangerfile just needs to be updated with the "xccov_file_direct_path" key to point at the filepath to the extracted file. Example below:

# Check code coverage
xcov.report(
   scheme: 'KillerApp',
   workspace: 'KillerApp.xcworkspace',
   minimum_coverage_percentage: 0.0,
   output_directory: './reports',
   xccov_file_direct_path: 'action.xccovreport'
)

Do note that this tool relies on you knowing the path of your XCResult file. Our cloud CI has a handy environment variable for that.

[Blackjacx]

Will there be an official version of xccov(-danger) that supports this without workarounds?

[abotkin-cpi]

@joshdholtz Any updates about xcov & xcov-danger's future given the repo ownership change from NakioStudio? https://twitter.com/nakiostudio/status/1171835206300504065

[joshdholtz]

@abotkin-cpi Yeah yeah, sorry! Its current transferred to my personal github account (as you can tell) but I will be transferring in into https://github.com/fastlane-community so that there is a good support team behind 😊

There will be some spin up time since I have never worked in this codebase but I will try to get all the support you need for xcresult/xcode11 and other things as fast as I can 💪

[joshdholtz]

And I'm pretty familiar_ with the new xcresult format. I just recently added support for it inside of trainer here - fastlane-community/trainer#34

So hopefully this isn't too complicated of a change 😊

[abotkin-cpi]

Sounds great! Any plans to expand out in fastlane a tool to extract all the files out of the xcresult like screenshots (which snapshot would likely want to take advantage of) in addition to the code coverage file? We've been using our own Swift CLI to do so in our CI but would be nice for one of the fastlane tools like trainer or snapshot to consolidate all the xcresult file extraction into one fastlane tool that snapshot & trainer could then depend on (as well as others like us to depend on to get at screenshots in our XCResult files).

[joshdholtz]

@abotkin-cpi No plans at the moment but also no objections 😊 TBH, I had no idea this xcresult stuff existed until last week but its been really fun to jump into and learn 😇 Are you able to provide me with an example of what you are thinking or what you have been using so that I have something to go off of? I can connect the dots easier when I see the end result 😉

[abotkin-cpi]

We've so far been using the Swift tool our intern @rsukumar-cpi worked on this summer (https://github.com/ChargePoint/xcparse) to do this. It's mostly just about getting down to the ActionTestActivitySummary which should have the array of attachments in ActionTestAttachment. Then using the filename & payloadRef's Reference ID, you can extract all the screenshots out with xcresullttool. Biggest thing is there is a whole lot of nesting with ActionTestMetadata, ActionTestSummaryGroups, etc. that you have to go through multiple xcresultool JSON retrievals before you get to the attachments. You can check out the code in here for the high-level (https://github.com/ChargePoint/xcparse/blob/master/Sources/xcparse/XCPParser.swift) & the coding/decoding stuff is all here (https://github.com/ChargePoint/xcparse/tree/master/Sources/XCParseCore).

Code coverage file is pretty straightforward, just need to grab the Reference ID off of the ActionRecord's ActionResult's CodeCoverageInfo's reportRef.

In terms of what would be nice in a fastlane tool, it'd be great to similarly have what xcparse is doing where it takes an xcresult path & will just export out the screenshot files in a given directory or get the code coverage file. That would allow other tools like snapshot, xcov, & trainer to take advantage of a centralized XCResult parser and not get into the business of having to know the XCResult format in each of the tools themselves. It would also help situations like our CI where we want to be able to keep all the PNG screenshots from UI automation as build artifacts that can be viewed on the web. Example from our CI step:

# Extract the screenshots from the XCResult
xcparse -s $BITRISE_XCRESULT_PATH .

# Copy the screenshots over into the deployment folder
find testScreenshots -type f -name 'CPTAutomation*.png' -exec cp {} $DEPLOY_DIR \;

Here's a snapshot of what the end result would be if we run a command to extract various files from the XCResult like screenshots (using -s in xcparse) or the code coverage (using -x in xcparse). Note the raw action.xccovreport that we can access (and which xcov & xcov-danger would want to get at) & the folder filled with all the screenshot attachments from the test runs in the XCResult

[joshdholtz]

@abotkin-cpi This is beautiful 🙌 Thanks for taking the time to write all that app and provide screenshots ❤️ I would love to be able to get as much of this working in fastlane as we can 💪 Cool if I hit you up if I have any questions on any of this as I start playing around with it?

[abotkin-cpi]

Yep, hit me up if you have any questions. My email should be visible on my GitHub profile & I do use Slack if you have any Slack workspace.

[joshdholtz]

@abotkin-cpi Got a PR out for adding Xcode 11 support into xcov ☝️ I currently don't have any great way to test a danger setup for this but if you (or somebody) else is able to test this that would be 💯 The PR I made should just work since I didn't need to add any new options or anything 🙃🤞

I think you can test this by...

Update your Gemfile to 👇 and run bundle install or bundle update xcov

gem 'xcov`, :git => 'https://github.com/fastlane-community/xcov.git', :branch => 'joshdholtz-xcresult-support'

[Blackjacx]

If you use this in your Gemfile it might work better (small typo above):

gem 'xcov', :git => 'https://github.com/fastlane-community/xcov.git', :branch => 'joshdholtz-xcresult-support'`

Cannot test this right now since we switched to danger-xcov last week. But yeah would be interested in a working solution too :-)

[chrisballinger]

I can confirm using the xcov fork fixes things without needing any additional configuration changes

[abotkin-cpi]

@joshdholtz I'll give this a try later this afternoon/evening with our CI setup & get back to you in 12 hours.

[chrisballinger]

@abotkin-cpi The xcov gem has a new official release that fixes this issue, you should be able to just do a simple bundle update and be good to go.

[abotkin-cpi]

My co-worker @mgorkani-cpi ran it & confirms this works.

One thing that is preventing us from using it in our CI is that our CI provider is placing the XCResult file outside of the derived data folder. When we extracted the code coverage file ourselves, we could used the xccov_file_direct_path argument, but now that we're experimenting with having xcov do the extraction itself, we hit the issue of xcov not providing a way to pass the path to an xcresult directly. I'll write a separate issue over in the xcov project about that.

EDIT: Issue regarding passing xcresult path directly to xcov - fastlane-community/xcov#156