Improve the documentation to explain usage of LLVM_PROFILE_FILE environment variable

[dbr]

I did cargo new --bin covtest and modified src/main.rs to contain this:

fn main() {
    if std::env::args().len() > 1 {
        println!("Hello, world!");
    } else {
        println!("Nope");
    }
}

I then, to the best of my knowledge, followed the instructions in the README (as of cf5c0a7)

$ cargo +nightly install grcov # unsure if the +nightly part is necessary here, seemed same either way
$ rustup component add --toolchain nightly llvm-tools-preview
$ export RUSTFLAGS="-Zinstrument-coverage"
$ cargo +nightly build
$ cargo +nightly run --
$ wc default.profraw 
  0   3 272 default.profraw
$ grcov . -s . --binary-path ./target/debug/ -t html --branch -o ./target/debug/coverage/

However the resulting report target/debug/coverage/index.html contains an empty table, as if no source files are found:

$ tree target/debug/coverage/
target/debug/coverage/
└── index.html

0 directories, 1 file

If I use the -Zprofile based, I do get output:

$ export CARGO_INCREMENTAL=0
$ export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort"
$ export RUSTDOCFLAGS="-Cpanic=abort"
$ grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o ./target/debug/coverage/
$ tree target/debug/coverage/
target/debug/coverage/
├── index.html
└── src
    ├── index.html
    └── main.rs.html

1 directory, 3 files

$ rustc +nightly --version
rustc 1.52.0-nightly (94736c434 2021-02-27)
$ grcov --version
grcov 0.7.1

..and this is on Debian Buster. Not sure what other information might be relevant - let me know if there's any other details I can provide. Thanks!

[tifrel]

I am having a something similar on OSX. @dbr, are your reports off (in terms of lines) when you use -Zprofile?
E.g. I get comment lines marked red.

❯ rustc +nightly --version
rustc 1.52.0-nightly (d6eaea1c8 2021-03-14)
❯ grcov --version
grcov 0.7.1

[dbr]

@TILLYBOY The -Z profile coverage is a bit off for me insomuch as things like some closing } lines or match ... { lines are marked as not covered (despite the surrounding lines being executed) - but I think that is to be expected, and it is why the source-based-coverage should be better (as it has better awareness of Rust's syntax)

[petreeftime]

I've tried versions 0.6.1 through 0.7.1 and they all produce an empty output. Might be an issue with nightly for instrument-coverage.

[dbr]

@petreeftime Ah that's a good point; using the nightly from the day this blog post was made (2020-11-12) produces same result, but using a random one from a month later (nightly-2020-12-12) seems to work

$ cat rust-toolchain 
nightly-2020-12-12
$ rustup component add llvm-tools-preview
..
$ export RUSTFLAGS="-Zinstrument-coverage"
$ cargo run
...
$ grcov . -s . --binary-path ./target/debug/ -t html --branch -o ./target/debug/coverage/
$ tree target/debug/coverage/
target/debug/coverage/
├── index.html
└── src
    ├── index.html
    └── main.rs.html

1 directory, 3 files

So I guess the question is if the problem lies with Rust (the output from newer nightlys is broken) or grcov (the output changed for some legitimate reason and it needs updated)?

[marco-c]

I can't reproduce this with grcov 0.7.1 and latest Rust nightly. Are you still able to reproduce?

[link2xt]

I can still reproduce this. I have recently added coverage-generating script to chatmail/core#2347 which uses non-source-based coverage because of this bug.

If I clone https://github.com/deltachat/deltachat-core-rust/ and try to generate source-based coverage, even with +nightly toolchain, it generates empty profile.

default.profraw generated is 16M, but HTML shows nothing is covered by tests.

[petreeftime]

I've updated to latest nightly and it seems to work for me now.

[marco-c]

@link2xt can you upload the default.profraw here?

[link2xt]

I am running:

git clean -xffd
git reset --hard
rustup update nightly
export RUSTFLAGS="-Zinstrument-coverage"
cargo +nightly test
grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o ./coverage/
firefox coverage/index.html

Here is what I get:

Profraw file attached, is it useful without the binaries though?
profraw.zip

[dbr]

nightly-2021-04-15 seems to be working for my simple test case, shall try in the larger project which initially caused the problem soon

[link2xt]

Could anyone who reports latest nightly as working for them try to generate source-based coverage for https://github.com/deltachat/deltachat-core-rust/ ? It does not require any additional setup besides git clone and cargo test to run the tests in the repository.

[dbr]

@link2xt I got the same result just running cargo test with the flag run, however I think it is because the default.profraw file is getting overwritten by multiple processes (one for the tests, one for doctests, etc)

https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html#test-coverage

If I run

LLVM_PROFILE_FILE="coverage-pid%p.profraw" RUSTFLAGS="-Zinstrument-coverage" cargo +nightly-2021-04-15 test

then I get three files, two produce empty coverage and the other produces valid looking result - it produces this following error (which is likely an unrelated problem), but still writes the coverage report:

$ grcov coverage-pid31325.profraw -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o ./coverage/
01:47:40 [ERROR] Execution count overflow detected.

[link2xt]

@dbr Separating by process ID helps indeed, I think the bug can be closed.

Although I found no way to get it down to one file. RUSTFLAGS="-Zinstrument-coverage" LLVM_PROFILE_FILE="coverage-pid%p.profraw" cargo +nightly-2021-04-15 test --tests still produces two files, one of which is empty.

[marco-c]

I mentioned this in the blog post (https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html) and in the CI section of the README.md, but I'll document this better in the How to generate source-based coverage for a Rust project section too.