New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate gcov coverage data #646

Open
steveklabnik opened this Issue Jan 21, 2015 · 22 comments

Comments

Projects
None yet
@steveklabnik
Member

steveklabnik commented Jan 21, 2015

Issue by brson
Thursday Jul 14, 2011 at 17:33 GMT

For earlier discussion, see rust-lang/rust#690

This issue was labelled with: A-an-interesting-project, A-debuginfo, A-testsuite in the Rust repository


It would be nice to know what our test coverage looks like. I believe this depends on rust-lang/rust#689

@reem

This comment has been minimized.

Show comment
Hide comment
@reem

reem Feb 14, 2015

Relevant hyper issue (linking): hyperium/hyper#44

reem commented Feb 14, 2015

Relevant hyper issue (linking): hyperium/hyper#44

@lifthrasiir

This comment has been minimized.

Show comment
Hide comment
@lifthrasiir

lifthrasiir Mar 14, 2015

Contributor

Kcov had an issue with position-independent executables that rustc generates by default. This should be fixed by a PR at SimonKagstrom/kcov#77. (EDIT: This is now merged, upcoming kcov v26 or later should be fine.)

Contributor

lifthrasiir commented Mar 14, 2015

Kcov had an issue with position-independent executables that rustc generates by default. This should be fixed by a PR at SimonKagstrom/kcov#77. (EDIT: This is now merged, upcoming kcov v26 or later should be fine.)

@farcaller

This comment has been minimized.

Show comment
Hide comment
@farcaller

farcaller Mar 15, 2015

Just for the reference, SimonKagstrom/kcov#77 allows to get coverage output from a casual binary produced by cargo build/test (on linux host only, obviously).

farcaller commented Mar 15, 2015

Just for the reference, SimonKagstrom/kcov#77 allows to get coverage output from a casual binary produced by cargo build/test (on linux host only, obviously).

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Jan 14, 2016

Recently a compiler-rt commit removed the explicit profile targets, and I'm not having any luck with just adding the required functions to FUNCTIONS.builtins in make/platform/triple.mk: compiler-rt builds and seems usable, but rustc segfaults when using -C passes=insert-gcov-profiling

shahn commented Jan 14, 2016

Recently a compiler-rt commit removed the explicit profile targets, and I'm not having any luck with just adding the required functions to FUNCTIONS.builtins in make/platform/triple.mk: compiler-rt builds and seems usable, but rustc segfaults when using -C passes=insert-gcov-profiling

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Jan 25, 2016

Turns out the segfault is fixed by rust-lang/rust#31176 - so this might again be a possibility to investigate!

shahn commented Jan 25, 2016

Turns out the segfault is fixed by rust-lang/rust#31176 - so this might again be a possibility to investigate!

@frewsxcv

This comment has been minimized.

Show comment
Hide comment
@frewsxcv

frewsxcv Jan 25, 2016

Member

@shahn Have you tried running insert-gcov-profiling after rust-lang/rust#31176 got merged?

Member

frewsxcv commented Jan 25, 2016

@shahn Have you tried running insert-gcov-profiling after rust-lang/rust#31176 got merged?

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Jan 25, 2016

Yes, but I don't have it working (yet, maybe?). Currently fighting with llvm (https://llvm.org/bugs/show_bug.cgi?id=26294)

shahn commented Jan 25, 2016

Yes, but I don't have it working (yet, maybe?). Currently fighting with llvm (https://llvm.org/bugs/show_bug.cgi?id=26294)

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Jan 27, 2016

So, I have it working locally. It requires a few things: A custom-built (nightly) rustc and compiler-rt with support for profiling, not using more than one codegen-units, adding the insert-gcov-profiling pass to the rustc invocation as well as linking with the custom compiler-rt.
I also had to symlink the stdlib source into my project's src directory because I couldn't figure out how to ignore these. Another issue happened around the lazy_static crate I used, because lcov really wanted a file <lazy_static macros> to exist in my project's top level directory.

The generated coverage data looks mostly ok, but has some strange things:

  • it reports all derives as unused
  • in all struct declarations, it reports the declaration of members as unused
  • it reports "pub mod xy" lines as unused

The picture gets a bit worse when looking at branch coverage, which seems basically unusable for Rust in its current state. use statements sometimes creates tons of branches, mostly in never-executed basic blocks, but sometimes even in executed basic blocks. No idea what's up there. Generally rust code seems to be very branch heavy.

shahn commented Jan 27, 2016

So, I have it working locally. It requires a few things: A custom-built (nightly) rustc and compiler-rt with support for profiling, not using more than one codegen-units, adding the insert-gcov-profiling pass to the rustc invocation as well as linking with the custom compiler-rt.
I also had to symlink the stdlib source into my project's src directory because I couldn't figure out how to ignore these. Another issue happened around the lazy_static crate I used, because lcov really wanted a file <lazy_static macros> to exist in my project's top level directory.

The generated coverage data looks mostly ok, but has some strange things:

  • it reports all derives as unused
  • in all struct declarations, it reports the declaration of members as unused
  • it reports "pub mod xy" lines as unused

The picture gets a bit worse when looking at branch coverage, which seems basically unusable for Rust in its current state. use statements sometimes creates tons of branches, mostly in never-executed basic blocks, but sometimes even in executed basic blocks. No idea what's up there. Generally rust code seems to be very branch heavy.

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Jan 27, 2016

Oh, one more thing. One cannot use gcc's gcov tool to interpret the data depending on gcc version, but rather llvm-cov (to be built from the make generated during rustc's build process) because the data format varies wildly between gcov versions. gcov from new GCC just segfaults when given such a gcda file.

What's the process to proceed here? I would love to add support for this to be easier and actually be eventually shipped by default with rustc (I'm thinking "cargo coverage" or something). This would need build system changes, I have absolutely zero idea about windows compatibility (it works on OSX and Linux), and also an idea how to distribute the necessary compiler-rt as well as llvm-cov (perhaps relabeled to rustcov or something to avoid name clashes with a concurrently installed llvm). Is it worthwhile to start the process of implementing the necessary changes?

shahn commented Jan 27, 2016

Oh, one more thing. One cannot use gcc's gcov tool to interpret the data depending on gcc version, but rather llvm-cov (to be built from the make generated during rustc's build process) because the data format varies wildly between gcov versions. gcov from new GCC just segfaults when given such a gcda file.

What's the process to proceed here? I would love to add support for this to be easier and actually be eventually shipped by default with rustc (I'm thinking "cargo coverage" or something). This would need build system changes, I have absolutely zero idea about windows compatibility (it works on OSX and Linux), and also an idea how to distribute the necessary compiler-rt as well as llvm-cov (perhaps relabeled to rustcov or something to avoid name clashes with a concurrently installed llvm). Is it worthwhile to start the process of implementing the necessary changes?

@nrc

This comment has been minimized.

Show comment
Hide comment
@nrc

nrc Jan 27, 2016

Member

@shahn Could you link to your branch please? It would be good to see what changes are necessary to rustc.

The ideal way forward for this is to have a crate which does it outside of the main Rust repo and to develop it there until it is a bit more mature. Deep changes to the compiler should land if needed, but a lot of stuff can be done by using the compiler as a library and wrapping calls to it. You can also make Cargo plugins to experiment with cargo coverage. If there is not much to do other than compiler changes then the next step would be to put a PR together and discuss on that. If it is a lot of changes, you could make a compiler RFC, but that is probably not necessary.

This sounds really cool! Thanks for investigating it.

Member

nrc commented Jan 27, 2016

@shahn Could you link to your branch please? It would be good to see what changes are necessary to rustc.

The ideal way forward for this is to have a crate which does it outside of the main Rust repo and to develop it there until it is a bit more mature. Deep changes to the compiler should land if needed, but a lot of stuff can be done by using the compiler as a library and wrapping calls to it. You can also make Cargo plugins to experiment with cargo coverage. If there is not much to do other than compiler changes then the next step would be to put a PR together and discuss on that. If it is a lot of changes, you could make a compiler RFC, but that is probably not necessary.

This sounds really cool! Thanks for investigating it.

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Jan 29, 2016

Yes, of course. Sorry for the delay, I was busy otherwise. Note that this is not even in pre-PR state, it is just enough to get the coverage build working for rustc on OSX.

shahn commented Jan 29, 2016

Yes, of course. Sorry for the delay, I was busy otherwise. Note that this is not even in pre-PR state, it is just enough to get the coverage build working for rustc on OSX.

@frewsxcv

This comment has been minimized.

Show comment
Hide comment
@frewsxcv

frewsxcv Feb 25, 2016

Member

@shahn Out of curiosity, do you remember why you removed the -std=c99 flag in the rustc commit? Also, after those changes, what did you do to actually generate coverage? Just run gcov?

Member

frewsxcv commented Feb 25, 2016

@shahn Out of curiosity, do you remember why you removed the -std=c99 flag in the rustc commit? Also, after those changes, what did you do to actually generate coverage? Just run gcov?

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Feb 25, 2016

the -std=c99 was added in a commit for msvc compat iirc (I'm not sure if it is required), and breaks the build for compiler-rt with coverage. I don't have a windows box to test this on. To actually generate the coverage, you need to:
RUSTC=$repo/$target/stage1/bin/rustc cargo rustc -- -C passes=insert-gcov-profiling -C link-args=$rustc/$target/rt/compiler-rt/triple/builtins/libcompiler_rt.a

where $rustc is your rust git checkout and $target is the target triple

shahn commented Feb 25, 2016

the -std=c99 was added in a commit for msvc compat iirc (I'm not sure if it is required), and breaks the build for compiler-rt with coverage. I don't have a windows box to test this on. To actually generate the coverage, you need to:
RUSTC=$repo/$target/stage1/bin/rustc cargo rustc -- -C passes=insert-gcov-profiling -C link-args=$rustc/$target/rt/compiler-rt/triple/builtins/libcompiler_rt.a

where $rustc is your rust git checkout and $target is the target triple

@frewsxcv frewsxcv referenced this issue Feb 29, 2016

Open

Track test coverage #132

1 of 5 tasks complete

@critiqjo critiqjo referenced this issue Mar 29, 2016

Merged

Add .travis.yml #5

@briansmith

This comment has been minimized.

Show comment
Hide comment
@briansmith

briansmith Jun 20, 2016

Consider this:

x = try!(foo());

I would like the code coverage to report which of the Ok and Err branches were considered. i.e. like coverage tools would do if macros were expanded. In general, every branch hidden by macros should be reported separately.

Similarly, in the case of the ? operator, every hidden branch of the ? should be reported separately.

briansmith commented Jun 20, 2016

Consider this:

x = try!(foo());

I would like the code coverage to report which of the Ok and Err branches were considered. i.e. like coverage tools would do if macros were expanded. In general, every branch hidden by macros should be reported separately.

Similarly, in the case of the ? operator, every hidden branch of the ? should be reported separately.

@whitequark

This comment has been minimized.

Show comment
Hide comment
@whitequark

whitequark commented Dec 25, 2016

@shahn ping?

@shahn

This comment has been minimized.

Show comment
Hide comment
@shahn

shahn Dec 25, 2016

@whitequark I linked the branches above, I wasn't ready to commit more because i still am not clear how it all fits together. What exactly are you pinging about?

shahn commented Dec 25, 2016

@whitequark I linked the branches above, I wasn't ready to commit more because i still am not clear how it all fits together. What exactly are you pinging about?

@whitequark

This comment has been minimized.

Show comment
Hide comment
@whitequark

whitequark Dec 25, 2016

@shahn I wondered if you did any more work on this--but nevermind, I already have an idea of how to proceed and will try to send some PRs shortly.

whitequark commented Dec 25, 2016

@shahn I wondered if you did any more work on this--but nevermind, I already have an idea of how to proceed and will try to send some PRs shortly.

@whitequark

This comment has been minimized.

Show comment
Hide comment
@whitequark

whitequark Dec 26, 2016

See rust-lang/rust#38608 writeup on how to generate useful information from this is incoming.

whitequark commented Dec 26, 2016

See rust-lang/rust#38608 writeup on how to generate useful information from this is incoming.

@marco-c

This comment has been minimized.

Show comment
Hide comment
@marco-c

marco-c Sep 19, 2017

rust-lang/rust#42433 (a rebase of rust-lang/rust#38608) was merged.
rust-lang/rust#42524 is tracking the stabilization of the feature.

marco-c commented Sep 19, 2017

rust-lang/rust#42433 (a rebase of rust-lang/rust#38608) was merged.
rust-lang/rust#42524 is tracking the stabilization of the feature.

@steveklabnik

This comment has been minimized.

Show comment
Hide comment
Member

steveklabnik commented Nov 8, 2017

@marco-c

This comment has been minimized.

Show comment
Hide comment
@marco-c

marco-c Nov 20, 2017

https://github.com/kennytm/cov

There's also https://github.com/marco-c/grcov, which is relying on LLVM to parse the GCOV files instead of reimplementing the parsing.

marco-c commented Nov 20, 2017

https://github.com/kennytm/cov

There's also https://github.com/marco-c/grcov, which is relying on LLVM to parse the GCOV files instead of reimplementing the parsing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment