Oddity in what lines count towards coverage

[Nokel81]

I asked this over on mozilla/grcov but then realized that it is better suited to here.

While using the following RUSTFLAGS to generate coverage reports I have found some odditites with what is considers to be covered/not-covered/not-needed to be covered.

-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Zno-landing-pads

1. Test functions (all of mod tests) are counted towards coverage
2. Some comments (including doc comments) are counted towards coverage (and sometimes not covered even though the "line" was passed through)
3. struct/enum definitions (even if that variant/type is used or created)
4. trait definitions (even if a type which implements that trait is used through a trait typed variable)
5. Close braces (even if both branches are taken)
6. use statements

Would it be possible to either add a flag to ignore these, or to just ignore them all the time?

[jonas-schievink]

Coverage is based on debug info. If there's a bug in debuginfo generation, please provide a more concrete example (like the code needed to reproduce the issue, and an explanation about why the emitted debuginfo is wrong).

[bjorn3]

1. Test functions (all of mod tests) are counted towards coverage

They are compiled, so they get debuginfo.

2. Some comments (including doc comments) are counted towards coverage (and sometimes not covered even though the "line" was passed through)

Could you give an example where it is and isn't counted?

3. struct/enum definitions (even if that variant/type is used or created)

If you derive a trait for a struct/enum then the impls will have the struct/enum definition as span.

4. trait definitions (even if a type which implements that trait is used through a trait typed variable)

Vtable shims?

5. Close braces (even if both branches are taken)

Likely dropping of locals.

6. use statements

I don't know why that is.

[AlexanderWillner]

I was wondering myself. To give a specific example from How to Write Tests:

pub fn greeting(name: &str) -> String {
    format!("Hello {}!", name)
}

pub fn add_two(a: i32) -> i32 {
    a + 2
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn greeting_contains_name() {
        let result = greeting("Carol");
        assert!(result.contains("Carol"));
    }

    #[test]
    fn it_adds_two() {
        assert_eq!(4, add_two(2));
    }

}

Executing

cargo clean
CARGO_INCREMENTAL=0 RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Copt-level=0 -Zpanic_abort_tests -Cpanic=abort" cargo b
CARGO_INCREMENTAL=0 RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Copt-level=0 -Zpanic_abort_tests -Cpanic=abort" cargo t
grcov target/debug/ --ignore "/*" -s . --llvm --branch --ignore-not-existing -o html -t html

Gives a 76.9% line coverage and 62.5% function coverage...

[wesleywiser]

I'm curious if folks here have used the instrumentation-based code coverage feature? As @jonas-schievink points out, grcov uses the debuginfo rustc generates to map executed instructions back to functions/lines but the process rustc uses to map debuginfo to lines is biased towards what is useful for debugging code, not what is useful for code coverage.

While it's possible we could make tweaks there, we already have a specific code coverage feature so I would encourage folks to use that instead of debuginfo-based code coverage tools. If there are tooling gaps with instrumentation-based code coverage, we should try to close them.

[marco-c]

@wesleywiser grcov supports both options. Source-based coverage is the suggested one in grcov's docs: https://github.com/mozilla/grcov#example-how-to-generate-source-based-coverage-for-a-rust-project.