Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always 0% #22

Closed
Streel88 opened this issue Apr 19, 2016 · 44 comments
Closed

Always 0% #22

Streel88 opened this issue Apr 19, 2016 · 44 comments

Comments

@Streel88
Copy link

Streel88 commented Apr 19, 2016

Everyone I run xcov I am getting 0%

  • I have code coverage enabled in the Test Config
  • Ensured the Scheme shared

I created a new Project with some quick code and Xcode is showing the count beside each function inside the IDE.

My Command
`xcov -s AutoTest -p ./AutoTest.xcodeproj -o ./build/reports/xcov_output

I get the following Output

[14:47:22]: xcrun xcodebuild -list -project './AutoTest.xcodeproj'
[14:47:23]: Skipping file blacklisting as no ignore file was found at path ./.xcovignore

+-----------------------------+------------------------------------------+
| Summary for xcov 0.7 |
+-----------------------------+------------------------------------------+
| scheme | AutoTest |
| project | ./AutoTest.xcodeproj |
| output_directory | ./build/reports/xcov_output |
| derived_data_path | /Users/christopher.streel/Desktop/Build/ |
| minimum_coverage_percentage | 0.0 |
| ignore_file_path | ./.xcovignore |
| include_test_targets | false |
| skip_slack | false |
+-----------------------------+------------------------------------------+

[14:47:23]: $ /Library/Ruby/Gems/2.0.0/gems/xcov-0.7/lib/xcov-core/bin/xcov-core -s /Users/christopher.streel/Desktop/Build/Logs/Test/42D0D29B-E7A5-4A7D-B5BF-36058F0AE62E.xccoverage -o /var/folders/wz/p18ghh6561v49_s7mgdsvhg1lyr51y/T/report.json20160419-48755-1yu5l3q
[14:47:23]: ▸ Loading...
[14:47:23]: ▸ ------ xCov-Core ------
[14:47:23]: ▸ Opening .xccoverage file at path: /Users/christopher.streel/Desktop/Build/Logs/Test/42D0D29B-E7A5-4A7D-B5BF-36058F0AE62E.xccoverage
[14:47:23]: ▸ Parsing .xccoverage file...
[14:47:23]: ▸ File successfully parsed
[14:47:23]: ▸ Serializing coverage report...
[14:47:23]: ▸ Report successfully serialized
[14:47:23]: ▸ Writing report on disk...
[14:47:23]: ▸ Coverage report successfully created at path: /var/folders/wz/p18ghh6561v49_s7mgdsvhg1lyr51y/T/report.json20160419-48755-1yu5l3q
+--------------+-----------+
| xcov Coverage Report |
+--------------+-----------+
| AutoTest.app | 0% |
+--------------+-----------+

I've attached a screenshot, if you need any more info let me know.

I love the html output, just hope the Cover Coverage gets corrected

screen shot 2016-04-19 at 2 53 12 pm

@Streel88
Copy link
Author

Streel88 commented Apr 19, 2016

So I found a difference between using xcodebuild and the IDE

So I get a correctly generated report if I run the test with the Xcode IDE, but if I use xcodebuild it produces nothing, what params need to be passed in with xcodebuild?

My current command is as follows
xcodebuild clean test -project ./AutoTest.xcodeproj/ -scheme AutoTest -destination 'platform=iOS Simulator,name=iPhone 5s,OS=9.3' -enableCodeCoverage YES | xcpretty --report junit --report html

Update:
When I edit the scheme and disable Debug Executable, it fails to generate a coverage report

+-----------------------------+-----------------------------+
| Summary for xcov 0.7 |
+-----------------------------+-----------------------------+
| scheme | AutoTest |
| project | ./AutoTest.xcodeproj |
| output_directory | ./build/reports/xcov_output |
| minimum_coverage_percentage | 0.0 |
| ignore_file_path | ./.xcovignore |
| include_test_targets | false |
| skip_slack | false |
+-----------------------------+-----------------------------+

[16:34:48]: $ /Library/Ruby/Gems/2.0.0/gems/xcov-0.7/lib/xcov-core/bin/xcov-core -s /Users/christopher.streel/Library/Developer/Xcode/DerivedData/AutoTest-ayuflxopsdabfycvkjxznehpvvip/Logs/Test/6914B9BE-D82A-46AB-BE66-1C95EC821CF8.xccoverage -o /var/folders/wz/p18ghh6561v49_s7mgdsvhg1lyr51y/T/report.json20160419-51816-vuau4
[16:34:48]: ▸ Loading...
[16:34:48]: ▸ ------ xCov-Core ------
[16:34:48]: ▸ Opening .xccoverage file at path: /Users/christopher.streel/Library/Developer/Xcode/DerivedData/AutoTest-ayuflxopsdabfycvkjxznehpvvip/Logs/Test/6914B9BE-D82A-46AB-BE66-1C95EC821CF8.xccoverage
[16:34:48]: ▸ Parsing .xccoverage file...
[16:34:48]: ▸ File successfully parsed
[16:34:48]: ▸ Serializing coverage report...
[16:34:48]: ▸ Report successfully serialized
[16:34:48]: ▸ Writing report on disk...
[16:34:48]: ▸ Coverage report successfully created at path: /var/folders/wz/p18ghh6561v49_s7mgdsvhg1lyr51y/T/report.json20160419-51816-vuau4
+--------------+-----------+
| xcov Coverage Report |
+--------------+-----------+
| AutoTest.app | 0% |
+--------------+-----------+

screen shot 2016-04-19 at 4 34 55 pm

@nakiostudio
Copy link
Collaborator

Hi @Streel88 👋

Try adding the build argument to your xcodebuild command. That should be the only difference between running the tests from Xcode and running them from the console. I hope it helps!

@Streel88
Copy link
Author

Streel88 commented Apr 19, 2016

Hi @nakiostudio

I tried that and still no luck, to see if you can replicate my issue

  1. Ensure your running Xcode 7.3
  2. Purge your Derived Data, either manually or via Xcode
  3. cd into your Projects directory
  4. Execute the following command: "xcodebuild build test -project ./Project.xcodeproj/ -scheme Project -destination 'platform=iOS Simulator,name=iPhone 5s,OS=9.3' -enableCodeCoverage YES
  5. Execute the next command xcov -s Project -p ./Project.xcodeproj/ -o ./build/reports/xcov_output

You should see the output as 0%

Note: I am seeing the exact same behaviour with Slather

Update:
You can see the exact same behaviour with the IDE if your disable Debugging

  1. Edit Scheme
  2. Disable Debugger (Test > Uncheck Debug Executable)
  3. Run Test with IDE
  4. Goto Report Navigator (cmd+8)
  5. View Code Coverage Report of the Test

Disable Debugging
screen shot 2016-04-20 at 10 23 09 am

No Code Coverage in IDE after disabling Debug Executable
screen shot 2016-04-20 at 10 25 16 am

@nakiostudio
Copy link
Collaborator

Hi again @Streel88 👋

You are absolutely right, the Debug Executable flag must be enabled in order to get the coverage information. Could you please try adding the -configuration "Debug" argument to your command?

@Streel88
Copy link
Author

Still no luck all that does is change the build configuration, from what I've learned about it is one of the executables it produces under ../CodeCoverage/.. needs to be executed which it doesn't seem to do with xcodebuild.

I will keep looking into it further to see if I can't figure out the correct command

@nakiostudio
Copy link
Collaborator

Honestly, I have no idea what's wrong... if you can publish a sample project with this problem so I can clone and test it. I haven't been able to repro this on my machine.

@Streel88
Copy link
Author

Sorry for the delayed reply, here is the dummy Project I put together to test
AutoTest.zip

Scenarios

  1. Test with IDE, code completion enabled. Run xcov -> Success
  2. Test with IDE, code completion enabled, Debug Disabled. Run xcov -> Fail (But expected since IDE also shows no coverage)
  3. Clean/Delete DerivedData Folder, Test with xcodebuild, Run xcov -> Fail

Commands
xcodebuild
xcodebuild build test -project ./AutoTest.xcodeproj/ -scheme AutoTest -destination 'platform=iOS Simulator,name=iPhone 5s,OS=9.3' -enableCodeCoverage YES
xcov
xcov -s AutoTest -p ./AutoTest.xcodeproj/ -o ./xcov_output

@redok016
Copy link

redok016 commented Jun 2, 2016

I have the same issue. Is there any updated information related with this?

@nakiostudio
Copy link
Collaborator

Hi @redok016 👋

I'm looking at the problem, I'll try to get a fix out soon.

Thanks @Streel88 for the additional information 👍

@mathcarignani
Copy link

I have the same issue with my UI tests, when I run scan (use xcodebuild) I don't have the coverage from those tests

@triage
Copy link

triage commented Jun 15, 2016

I had this issue till recently. We have a mostly objective-c project that we've only recently started adding swift to. All of my tests (mostly obj-c) would run, yet it appeared that only swift files were used in the computation of coverage reports -- leading to low percentages like 7%.

Turned on this build setting and now my objective-c files are included in coverage reports properly ... which was not a part of the readme instructions for xcov:
screen shot 2016-06-15 at 7 17 06 am

@thelvis4
Copy link
Contributor

thelvis4 commented Jul 12, 2016

After some investigations, here is what I've noticed:

  • Running tests from Xcode IDE differs from running tests using xcodebuild:

Comparing Logs/Test/XXX/Session-<some_uid>.log:

Xcode:

Launching with Xcode.DebuggerFoundation.Launcher.LLDB
...
"wait_for_debugger" = 1;

xcodebuild:

Launching with Xcode.IDEFoundation.Launcher.PosixSpawn
...
 "wait_for_debugger" = 0;
  • LauncherIdentifier depends on Debug Executable flag (in Xcode scheme):

Debug Executable enabled:

selectedDebuggerIdentifier = "Xcode.DebuggerFoundation.Debugger.LLDB"
selectedLauncherIdentifier = "Xcode.DebuggerFoundation.Launcher.LLDB"

Debug Executable disabled:

selectedDebuggerIdentifier = ""
selectedLauncherIdentifier = "Xcode.IDEFoundation.Launcher.PosixSpawn"

So, when running tests using xcodebuild, it behaves as when Debug Executable is disabled, no matter what the actual state of Debug Executable flag is.

  • When Debug Executable flag is disabled, Unit tests give correct coverage data while UI Tests do not.

Running Unit tests with Xcode.IDEFoundation.Launcher.PosixSpawn Launcher Identifier (similar with having Debug Executable flag disabled) gives proper coverage data. Running UI Tests with these configurations gives invalid coverage data (0% for Target application).
I suppose the differences come from different mechanisms used for starting UI Tests and Unit Tests.
Screenshot from a WWDC '16 presentation
WWDC '16 Advanced Testing and Continuous Integration video

P.S. If we could figure out how to run UI Tests using Xcode.DebuggerFoundation.Launcher.LLDB Launcher Identifier, I think we would get valid coverage data. So far, I couldn't find any info about how to do this.

@nakiostudio
Copy link
Collaborator

Great stuff @thelvis4 👍 I've been digging a bit into it and there is very little information. I will test a couple of things during the weekend though.

Thanks!

@redok016
Copy link

@thelvis4,@nakiostudio thanks for your great effort

@raven
Copy link

raven commented Jul 27, 2016

@thelvis4 have you filed a radar? Just wanting to dupe it if you have one already :)
Also, have you had a chance to check if the behaviour is different with Xcode 8?

@thelvis4
Copy link
Contributor

@raven Sorry, haven't had the time to further investigate (with Xcode 8) and to file a radar. But I would be glad to dupe the radar if you could file one. I believe we should check this on Xcode 8, as I doubt they will release another version of Xcode 7.

@stevepeak
Copy link

At Codecov we have resolved this issue with the help of customers. It appears that xcpretty corrupted the coverage reports. https://github.com/codecov/example-swift/issues/14#issuecomment-249427948

Hope this is related and find it useful. Thank you!

@nakiostudio
Copy link
Collaborator

Hi @stevepeak 👋

Thanks for the heads up, however the issue we are facing is pretty different. xcov processes swift 3 coverage reports without any problems. In addition, xcov works directly with .xccoverage files therefore it has nothing to do with xcpretty 😞

Thank you once again and keep up the good work with Codecov!

@iwllyu
Copy link

iwllyu commented Oct 31, 2016

I created a project that demonstrates this bug

https://github.com/iwllyu/UITestCoverageBug

If you run the tests via XCode IDE you get 80% of ViewController.swift coverage. If you run the tests via xcodebuild CLI you only get 40% coverage for ViewController.swift.

This was done with Xcode 7.3.1 - but I read that it's the same for XCode 8. I can test tonight when I get on my home machine. I'll file a radar if I confirm it on XCode 8

I filed a radar and also submitted to open rader here - https://openradar.appspot.com/29033348

not sure what you guys mean by "dupe", hopefully this is what you meant!

@thelvis4
Copy link
Contributor

thelvis4 commented Nov 2, 2016

@iwllyu Thank you for submitting the radar and putting it on open radar as well. 👍
I'm going to duplicate it.
btw, we use "dupe" as a short form of "duplicate".

@nakiostudio
Copy link
Collaborator

Thank you @iwllyu! I will take a look this weekend. Hopefully we will come up with a final solution for this issue.

@EliseLeb
Copy link

Has a solution been found?

@iwllyu
Copy link

iwllyu commented Dec 16, 2016

I got this on on 12/16/2016 @ 3:17AM

Engineering has determined that your bug report (29033348) is a duplicate of another issue (28900146) and will be closed.

The open or closed status of the original bug report your issue was duplicated to appears in the yellow "Duplicate of XXXXXXXX" section of the bug reporter user interface. This section appears near the top of the right column's bug detail view just under the bug number, title, state, product and rank.

An example of the duplicate section from the bug reporter user interface with your bug and the duplicate bug info is included below:

29033348 Test Coverage reports from xcodebuild CLI ignore coverage from UI tests

State: Closed Product:
Rank: No Value

Duplicate of 28900146 (Open)

@michaellady
Copy link

michaellady commented Dec 17, 2016

@iwllyu thank you for helping me retain my sanity!

@thelvis4
Copy link
Contributor

@EliseLeb This is definitely an inconsistency (bug) in Apple's tools. And it's still reproducible with Xcode 8.2.

Xcode IDE and xcodebuild use different UITests launchers (and debuggers):

  • Xcode IDE uses this plugin Xcode.app/Contents/PlugIns/DebuggerLLDB.ideplugin
    whereas
  • xcodebuild uses IDEFoundation framework Xcode.app/Contents/Frameworks/IDEFoundation.framework (and does't attach any debugger)

I couldn't make (force) xcodebuild to take Xcode.DebuggerFoundation.Launcher.LLDB as a launcher, although I tried half of a day in different undocumented (and probably stupid) ways. Probably it's an implementation detail and cannot be influenced externally.

At this point, I will wait for Apple to fix this bug/inconsistency. BTW, my radar was also marked as Duplicate of 28900146 (Open).

@EliseLeb
Copy link

@thelvis4 Thanks for the update! I also filed a bug with apple but not holding my breath. What a bummer for UI tests.

@earltedly
Copy link

I've filed a radar too - thanks everyone in this thread for saving me a few extra hours working out what was going on! Has anyone tried in the latest beta yet?

@thelvis4
Copy link
Contributor

@neonichu told me that the ticket is under investigation. 🤞

@earltedly
Copy link

@thelvis4 @neonichu it also might relate to rdar://problem/28571841 which I filed about UI tests always reporting 0% coverage for linked frameworks. There's a simple example project attached.

@michaellady
Copy link

Anyone try this on 8.3.2 yet?

@iwllyu
Copy link

iwllyu commented Apr 20, 2017

@michaellady you're welcome to clone the project here https://github.com/iwllyu/UITestCoverageBug and try it out on 8.3.2

@michaellady
Copy link

Hi @iwllyu, I'm still seeing the command line issue on 8.3.2

@iwllyu
Copy link

iwllyu commented Apr 21, 2017

thanks, I've updated the readme in that project

@JohnCoates
Copy link

I looked into fixing this. Found the difference in logs that @thelvis4 did. Figured I'd see if there was an undocumented flag I could pass to xcodebuild to use the LLDB launcher instead of PosixSpawn. Decompiling didn't find anything, so I decided to force it myself to see if it would work. Modifying xcode build to use LLDB instead of PosixSpawn indeed works, but its fragility won't be an ideal solution to implement into most projects. Most of us are probably stuck waiting for Apple to implement a fix, so file a Radar!

Proof of Concept commands:

# we'll run xcodebuild through a debugger so we can alter its execution
lldb

# set our debugging target
target create /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild

# launch process with arguments and stop at entry point
process launch -s -- -sdk iphonesimulator -destination 'name=iPhone SE' test

# set a breakpoint on an argument where we can alter the process launcher & debugger
# the method won't be loaded into the process yet, so don't worry about the warning lldb gives, the breakpoint will be resolved automatically as soon as the method is available
breakpoint set -F "+[IDELaunchParametersSnapshot launchParametersWithSchemeIdentifier:launcherIdentifier:debuggerIdentifier:launchStyle:runnableLocation:debugProcessAsUID:workingDirectory:commandLineArgs:environmentVariables:architecture:platformIdentifier:buildConfiguration:buildableProduct:deviceAppDataPackage:allowLocationSimulation:locationScenarioReference:showNonLocalizedStrings:language:region:routingCoverageFileReference:enableGPUFrameCaptureMode:enableGPUValidationMode:debugXPCServices:debugAppExtensions:internalIOSLaunchStyle:internalIOSSubstitutionApp:launchAutomaticallySubstyle:]"

# now let's add some commands to run when the breakpoint hits
# we'll replace the launcher & debugger arguments passed to the function and then continue execution
break command add
po $rcx = (unsigned long)IDEDefaultLauncherIdentifier
po $r8 = (unsigned long)IDEDefaultDebuggerIdentifier
continue
DONE

# resume execution from the entry point. coverage should be correct now
continue

Proof of concept code for injection as a library: https://gist.github.com/JohnCoates/605319ab31c0aa05c77973481aeae4b2

@iwllyu
Copy link

iwllyu commented Jun 13, 2017

https://github.com/coryalder is reporting that this is fixed in XCode9

iwllyu/UITestCoverageBug#4

@michaellady
Copy link

michaellady commented Jun 13, 2017 via email

@nakiostudio
Copy link
Collaborator

Amazing work by the dev tools team in this release 👏

@asos-petergoldsmith
Copy link

OMG

@iwllyu
Copy link

iwllyu commented Jul 24, 2017

Oops we may have jumped the gun.

iwllyu/UITestCoverageBug#4 (comment)

tl;dr: it looks like both xcode and xcodebuild are not covering UI tests. xcode could have erroneously reporting coverage before and Apple intends for UI tests not to be part of code coverage.

or it's a bug

Well, at least they're consistent now

@Vkt0r
Copy link

Vkt0r commented Sep 27, 2017

@iwllyu Any update in this?? I'm running Xcode 9 and after ran all the tests the UI Tests are missed from the xcoverage report. Xcov is only reporting the unit tests.

@vgorloff
Copy link

Xcode Version 9.3 beta (9Q98q) seems resolves issue with missed UI Tests coverage as long as Debug executable not enabled in Test schema. Details: https://stackoverflow.com/a/48477190/1418981

@raven
Copy link

raven commented Feb 12, 2018

Xcode Version 9.3 beta (9Q98q) seems resolves issue with missed UI Tests coverage as long as Debug executable not enabled in Test schema. Details: https://stackoverflow.com/a/48477190/1418981

I've verified this is indeed working again 🎉

@stephen-oconnor-cp
Copy link

Sorry to comment on this after so long, but this is the only issue that relates to the problem I'm having.

Not even unit tests are running right now. If I run them in Xcode, it generates coverage and does everything as expected. If it's run via run_tests / scan, it doesn't appear that unit tests are even running, even though it has output like this:

[10:13:55]: Running Tests: ▸ Touching CameraDummyTests.xctest (in target 'CameraDummyTests' from project 'CameraDummy')
[10:13:55]: ▸ Processing empty-BuildPipeSandbox.plist
[10:13:55]: ▸ Processing empty-BuildPipeSandboxTests.plist
[10:13:56]: ▸ Linking BuildPipeSandboxTests
[10:13:56]: Running Tests: ▸ Touching BuildPipeSandboxTests.xctest (in target 'BuildPipeSandboxTests' from project 'BuildPipeSandbox')
[10:13:56]: Running Tests: ▸ Touching BuildPipeSandbox.app (in target 'BuildPipeSandbox' from project 'BuildPipeSandbox')
[10:13:56]: ▸ Linking BuildPipeSandboxUITests
[10:13:56]: ▸ Processing empty-BuildPipeSandboxUITests.plist
[10:13:56]: ▸ Copying Info.plist
[10:13:56]: Running Tests: ▸ Touching BuildPipeSandboxUITests.xctest (in target 'BuildPipeSandboxUITests' from project 'BuildPipeSandbox')
[10:13:56]: ▸ Test build Succeeded
[10:13:56]: Copying .xctestrun file

Can anybody help with this?

@michaellady
Copy link

michaellady commented Jan 26, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests