[SR-1281] Code coverage testing with custom toolchains in Xcode doesn't work

[swift-ci]

Previous ID	SR-1281
Radar	None
Original Reporter	rfdickerson (JIRA User)
Type	Bug

Attachment: Download

Environment

swift-DEVELOPMENT-04-12-a, Mac OS X 10.11

Additional Detail from JIRA

Votes	3
Component/s	Compiler
Labels	Bug, CodeCoverage
Assignee	None
Priority	Medium

md5: 832192ff9d7246fb2eec64e5038f4b79

Issue Description:

When I add Code Coverage testing to the XCode project generated with 'swift build -X' then following error occurs:

I have also tried adding emiting profiling and coverage to the project with

`swift build -Xcc -fprofile-instr-generate -Xcc -ftest-coverage`

But that did not work, either

[belkadan]

I don't think we support code coverage generation with custom toolchains, because the version of LLVM used in the toolchain is out of sync with the version in Xcode. @vedantk, anything to add?

[belkadan]

I suppose Xcode could provide a better error message. Mind filing a Radar?

[swift-ci]

Comment by Robert F. Dickerson (JIRA)

You are right regarding the wrong version of llvm being used. I can get a default.profraw file from building with:

swift build -Xswiftc -profile-generate -Xswiftc -profile-coverage-mapping

Next, I run the tests

swift test

Next, using the LLVM branch used in the swift compiler, I can run

/Ninja-ReleaseAssert/llvm-macosx-x86_64/bin/llvm-profdata merge -o testprof.profdata default.profraw

The next step of viewing the data using llvm-cov is not working for me.

[ddunbar]

This is unfortunate for SwiftPM's Xcode project generation because it blocks us from enabling code coverage by default in the generated schemes (since we expect many of users to frequently be using toolchains, at least for now).

[hamishknight]

We ought to be able to use the llvm-profdata and llvm-cov binaries in the custom toolchain, as they are included in development snapshots. Currently Xcode 14 has a different failure mode for this, it just doesn't show the coverage in the editor. We're tracking this internally as rdar://25754212.