Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexecuted code that lives in root should display as 0% coverage #298

Closed
aparks5 opened this issue Mar 12, 2019 · 9 comments

Comments

@aparks5
Copy link

@aparks5 aparks5 commented Mar 12, 2019

i had to run the following to show files that aren't touched at all by my tests, but definitely live in the code base and are used in the primary client program.

i realize that without a *gcda, gcovr has nothing to go on, but its easy enough to find the missing files, why can't they all be marked and displayed as 0% and penalize our code coverage score?

ls -1 *gcda > covered.txt
ls -1 *gcno > allgcov.txt
cat covered.txt | cut -d. -f1 > covered.log
cat allgcov.txt | cut -d. -f1 > allfiles.log
diff covered.log allfiles.log 
diff covered.log allfiles.log > no_gtest_coverage.diff
cat no_gtest_coverage.diff ```
@latk

This comment has been minimized.

Copy link
Member

@latk latk commented Mar 12, 2019

Gcovr does process .gnco files when no .gcda files were generated to support exactly your use case.

If that isn't happening, there might be some problem, presumably with filters. The commands you've shown are not really helpful since they just compare file lists, no actual coverage reports. Could you instead

  • Tell me your gcovr --version and platform/OS?
  • Show the output when running gcovr in --verbose mode?
@aparks5

This comment has been minimized.

Copy link
Author

@aparks5 aparks5 commented Mar 12, 2019

gcovr 4.1
$ uname -a
Linux hostname 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
gcovr --object-directory="(full path redacted)/i386-gcov/" -r="(full path redacted)" --html-details -o index.html -s --fail-under-line 40 >> coverage_status.json

i can't show --verbose output due to the nature of my work, but as an example, here's the output of one of the gcov commands being executed:

gcov (file) --branch-counts --branch-probabilities --preserve-paths --object-directory (objdir)
*gcda:cannot open data file, assuming not executed
File 'file.cxx'
No executable lines
No branches
No calls
Removing '(file).gcov'```

if i add --keep to gcovr it will keep the gcov files but not include them in the html output

@latk

This comment has been minimized.

Copy link
Member

@latk latk commented Mar 13, 2019

I really cannot reproduce that. Which compiler (gcc or clang) and which version are you using? When I run a test case where there is only a .gcno file and no .gcda file, gcov will produce output like:

$ gcov /path/to/gcovr/tests/nested/subdir/A/file7.gcno --branch-counts --branch-probabilities --preserve-paths --object-directory /path/to/gcovr/tests/nested/subdir/A
/path/to/gcovr/tests/nested/subdir/A/file7.gcda:cannot open data file, assuming not executed
File 'subdir/A/file7.cpp'
Lines executed:0.00% of 2
No branches
No calls
Creating 'subdir#A#file7.cpp.gcov'

And gcovr will happily use the resulting gcov file. I'm not sure why in your case gcov is deleting the file instead. Note that gcovr's --keep option has no effect on the gcov tool.

I understand that you can't show your code, but without some kind of reproducible example it will be very hard to identify the problem. Things I can think of:

  • you are using a compiler that gcovr doesn't directly support (for best results, use GCC 5 to 7)
  • the uncovered file in question really doesn't contain any executable code, for example if it only contains declarations.
@aparks5

This comment has been minimized.

Copy link
Author

@aparks5 aparks5 commented Mar 13, 2019

I'm using GCC5.4
the file in question has executable code

the problem is that the object directory and the source directory differ. even including all the subdirs in the search path does not resolve this issue.

i see an old issue which was resolved but the fix is a hack to create a file that includes all the headers not executed, which is not really maintainable with a growing codebase #33

i will try to produce a minimal working example and post it


the issue i believe i have is separate build and source directories, which cannot be changed for my project (company-wide structure)

also related issue: #154

@latk

This comment has been minimized.

Copy link
Member

@latk latk commented Mar 13, 2019

Directory layout issues are very likely not at play here. This seems to be a gcov thing.

I have just created & observed a test case with an uncovered file that works fine on GCC 5.5.0 / Ubuntu but fails to show the uncovered file under GCC 5.3.0 / MinGW-W64 (Windows). In that sense, I managed to confirm both that gcovr works correctly and that the bug exists.

I am not sure what can be done here.

  • It might be possible to extract the source file name from gcov stdout output and display empty coverage data in the reports. However, this would skew coverage metrics, and more importantly: this would be indistinguishable from files that genuinly have no coverage. I don't think I will do that.

  • I can add a paragraph about these issues to the documentation (I'm leaving this issue open as a reminder).

Besides that, this seems to be an upstream bug in some versions of gcov. Users can avoid this by upgrading GCC. I think this bug was fixed in GCC Revision 239413 on 2016-08-12 (gcc-mirror/gcc@6d5f72f). I would expect all later releases (5.5+, 6.2+, 7+) to include the fix.

@aparks5

This comment has been minimized.

Copy link
Author

@aparks5 aparks5 commented Mar 13, 2019

thank you for your thorough investigation!

to provide some context, the bigger picture here is that, for a large legacy code base with few existing tests, having uncovered files displayed in HTML demonstrates to stakeholders the extent to which code is largely untested.

this hopefully puts some hard numbers on the extent to which code is untested, and also provides a sense of progress as classes are put under test.

as a back of the envelope calculation i used find . -name "*.cxx" | wc -l | head -n 1 to estimate the total lines of code (i realize with comments and whitespace this probably overestimates), and estimated coverage based on true coverage = gcov estimate / LOC from bash.

@latk latk closed this in fe2f770 Mar 14, 2019
@latk

This comment has been minimized.

Copy link
Member

@latk latk commented Mar 14, 2019

Ok, I wrote an FAQ entry about this problem (will be deployed to the website with the next release).

You might be interested in the sloccount tool (available in the Ubuntu repositories) for a more accurate line count. That will still underestimate your coverage (declarations vs executable statements, lines vs statements), but it would be far more accurate than wc -l.

@aparks5

This comment has been minimized.

Copy link
Author

@aparks5 aparks5 commented Mar 14, 2019

As a note for people, I would really advice against upgrading your local machine's gcc version if your team uses a complex and fragile toolchain. I spent hours undoing the damage I unleashed (i.e. not being able to build anything from our codebase) by upgrading gcc to try to resolve this issue. Next time I try something like that I'm going to use a VM.

@latk

This comment has been minimized.

Copy link
Member

@latk latk commented Mar 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.