Implement support for LLVMs code coverage instrumentation

[gnzlbg]

There are ways to more or less easily obtain code coverage information from rust binaries, some of which are in widespread use (e.g. travis-cargo + gcov + coveralls.io). However, these are either platform specific (gcov/kcov are linux only), or incomplete (coverage for documentation tests is not collected).

It would be better if rustc would be able to instrument rust binaries and tests using LLVM Coverage Instrumentation. This would allow code coverage information to work portably across the supported platforms, as well as produce reliable coverage information.

Ideally, such support would be integrated in an easy to use cargo coverage subcommand and the Rust Language Server, such that IDEs can use this information to guide the user while writing tests.

[alexcrichton]

I know I'd love to see this work! Do you know what it would entail in terms of what the hoops a prototype would have to jump through? Also, does this work reliably on platforms like Windows? Or is it still best on Linux and "works mostly elsewhere"?

[gnzlbg]

A prototype would need to generate the LLVM IR for the instrumentation, for that it needs to generate a mapping between source ranges and performance counters (I would start with just functions). Then it needs to generate and embed the IR in the object files. It would be a good idea to look how clang then outputs this into a file to use the same format (which is documented), and test that we can read it with llvm-cov in linux and macos (don't know about windows support but since clang is gaining full windows support if its not there it will be there soon).

I think that would be enough for a prototype, from there we can move on to generating instrumentation for conditionals (match/loop/jumps/ifs...) and blocks (how many iterations was a loop body executed). We could go down to expressions, but then it would be wise to offer users a way to control how much instrumentation is generated: functions, branches, and loops (which are branches) is typically enough. We should then support skipped regions (for conditional compilation), expansions (for macros, maybe plugins), and dealing with unreachable code (there is a performance counter that is always zero for that).

The meat of the work is in the source=> performance counter mapping, generating the IR, and generating the conforming output. llvm-cov works at least on mac and linux, but IIRC clang has options to generate coverage information even for particular gcov versions. The source code for all this is available so taking a look wouldn't hurt.

Also, does this work reliably on platforms like Windows? Or is it still best on Linux and "works mostly elsewhere"?

It works on Mac and Linux, don't know about Windows. Even if it doesn't work everywhere, this is probably the way to make it work in as much platforms as possible anyways. Clang support on windows is pretty good already and it is only getting better.

[alexcrichton]

@sujayakar's done some awesome work to prototype this on OSX at least, discovering:

• For now, codegen-units probably has to be one to make this work (worth investigating)
• The pass to insert into LLVM is -C passes=insert-gcov-profiling
• The runtime support is available through -C link-args=-lclang_rt.profile_osx, and that library
• This runtime support lives around the directory xcrun --find clang in a path like usr/lib/clang/7.3.0/lib/darwin/libclang_rt.profile_osx.a.
• Binaries run with that instrumentation will produce a .gcda file
• Next to xcrun --find clang, there's an llvm-cov tool, and that can be run over the output file to generate information.

That should at least get code coverage working on OSX in some respect! THis is also a pretty strong case that it may not be too hard to actually get this working for all platforms if LLVM's standard tool suite "just works".

[sanxiyn]

@alexcrichton That's gcov coverage, which is completely different from what @gnzlbg meant in this issue.

The correct pass name is instrprof, implemented in lib/Transform/Instrumentation/InstrProfiling.cpp. See also LDC LLVM profiling instrumentation.

[gnzlbg]

@sanxiyn is correct.

I just want to add that I don't think we should implement clang-like gcov coverage generation.

LLVM format can be used for many more things (e.g. profile guided optimizations), and LLVM's llvm-cov tool can generate gcov files from it.

[frewsxcv]

afaik, there's a PR open for this: #38608

[sanxiyn]

As far as I can tell, #38608 is still gcov coverage and not instrprof.

[alexcrichton]

I believe this is now implemented as -Z profile and tracked at #42524, so closing in favor of that.

[sanxiyn]

This is not implemented yet. LLVM coverage is different from gcov coverage, which is also different from sanitizer coverage. LLVM implements three separate coverage mechanisms.

[mehcode]

Would LLVM coverage ( instrprof ) enable usable code coverage of generic heavy code? None of the existing code coverage solutions work well with heavy use of generics.

My project (a gameboy emulator) makes very heavy usage of generics and it'd be wonderful to get code coverage working on integration tests but I'm not really sure how.

[gnzlbg]

I think good code coverage support is the single most important piece of tooling missing in Rust.

When I run cargo test, and everything is green, I get a good feeling, but this feeling actually means nothing at all. To make cargo test mean something, it would need to at least tell me how many code-paths of my application have been exercised.

This information does not mean much either (just because a code-path was exercised does not mean that it was exercised for all inputs), but it would mean more than nothing, and it would make writing tests and asserting the value of test way easier.

IMO adding this kind of capability to cargo test (and yes, it should be part of cargo test, people should always know their code coverage) would be extremely valuable.

There is only another part of tooling infrastructure that would come close to this in value, and that would be an undefined behavior detector for all rust code.

This might sound like a rant, but I just want to raise awareness that this is a very important issue at least for me (and from the other issues being filled, for others as well) because it directly affects the correctness of rust programs (we don't want them to only be memory safe, but also to have correct logic).

Maybe if rustfmt, clippy, and the RLS advance enough during the impl period, the Rust tool team could prioritize these two tools next?

If I don't have precise auto-completion information or my source code is not perfectly formatted, well, I can live with that. But if my program has undefined behavior and I am not catching it because I am only testing 40% of my code-paths then... I am screwed. Does this make sense?

[mssun]

This is not implemented yet. LLVM coverage is different from gcov coverage, which is also different from sanitizer coverage. LLVM implements three separate coverage mechanisms.

Hi, can anyone explain that what's the difference between "LLVM coverage" (profile-instr-generate) and "gcov" (insert-gcov-profiling)? I don't know the detailed implementations of these tow passes, but will profile-instr-generate be more accurate than insert-gcov-profiling.

[mssun]

I found @kennytm's answer to my question. Let me put here in case someone like me gets lost in the overwhelmed references: #44673 (comment)

[bossmc]

I've done a bit of prototyping on this to see what would be involved in using instrprof for coverage, and it's revealed (to me) that the coverage aspect of this feature is secondary to the "Profile Guided Optimization" feature that instrprof was originally written for.

So, here's a vague "plan of action" (with progress so far) to enable PGO and LLVM-assisted coverage:

• Add frontend-guided instrumentation to IR
  • Unstable
• Enhance the profile-rt library to dump this out
• Enable Profile-Guided Optimization passes
  • Unstable
• Create a coverage map
• Integrate LLVM coverage tooling into cargo

Add Frontend-Guided Instrumentation

This is currently implemented in its most basic form as -Zpgo-gen which adds the -pgo-instr-gen and instrprof passes to LLVM to add instrumentation of all functions and basic blocks (loop bodies, if statements etc.).

There may be other places that it's useful to add instrumentation (e.g. adding vtable-downcast counts to enable -pgo-icall-prom passes - which turn dynamic dispatch into static dispatch in fast paths).

👍

Enhance the profile-rt library

With the above work complete, the generated binaries will count (in-memory) each time an instrumented code path is hit. To extract this information, the profile-rt library should walk the various __llvm_pfr_X sections of the binary/memory and dump the values out to a file (Clang dumps to default.profraw). The format of this file is (as far as I can see) "documented" in LLVM/Clang's codebase. On the other hand, the current implementation of Rustc's profile-rt library is taken wholesale from LLVM and thus already supports this function.

👍

Enable profile-guided optimization

With a default.profraw, one can create a profdata file (using llvm-profdata merge). This file can be passed to the pgo-instr-use (and other) passes to enable optimisations based on gathered profile information. Again, this is supported in nightly Rustc today (through -Zpgo-use).

👍

Create a coverage map

LLVM's coverage tooling can use a "Coverage Map" to map between instrumentation counters and blocks of code. The format of the map is well documented at http://llvm.org/docs/CoverageMappingFormat.html and there's code in LLVM (http://llvm.org/doxygen/classllvm_1_1coverage_1_1CoverageMappingWriter.html) for creating the magic format. Note that this may require Rustc manually inserting instrumentation (rather than using -pgo-instr-gen) since it needs to know which counter corresponds to which code block. The code in clang (https://github.com/llvm-mirror/clang/blob/master/lib/CodeGen/CodeGenPGO.cpp) that does this is probably a good place to start.

Note that the LLVM coverage map format supports various useful features, not well handled in GCOV-style coverage:

• Column as well as line-number analysis (useful for inline expressions)
• Macro-expansion analysis (coverage can "see through" macros)

❌

Integrate coverage tooling into Cargo

Creating a cargo coverage (or, possibly better, just creating coverage from cargo test) by adding profiling to the test cargo-profile and running the resulting profile data through llvm's tooling would be really cool. Especially if the output format is compatible with tools like Coveralls.io and can produce useful graphical outputs for analysis afterwards.

Likely requires stabilising -Zpgo-gen or at least exposing enough of it that cargo can create a profiled build.

❌

---

Unanswered Questions

There are some bits of Rust code for which coverage is non-obvious:

• Do landing pads report coverage? If they do then we run into the same issue as #55352
  • Maybe we add instrumentation for them (to determine how often they're hit) but don't add those counters to the coverage map?
• How do we handle non-instantiated generics? If a generic is never instantiated then no code is created for it so there's no counters for it.
  • Presumably we need to ensure that the coverage map includes the counters that "would be there if there was an instantiation" then coverage will spot that the generic is not covered.
  • This applies to "default implementations" in traits and likely is impacted by specialisation too.
• Similarly, should separate instantiations of the same generic be covered separately? If I test Foo<i32> in my UTs but use Foo<String> in my real code, have I really tested my code (in this case, String needs drop so maybe there's a subtlety there)?
  • Create coverage map entries for all instantiations (as well as the "generic" counter entry from the previous bullet)? I don't think llvm-cov knows how to handle this in the coverage map at the moment, but it might make sense for it to do so to support this for C++ templates too?

[bossmc]

• Proc-macros (and the built-in #derives) need care too - the expanded code coverage is potentially uninteresting (if you trust the proc-macro) or critical (if you don't).
• Basically anywhere where rustc does "codegen" above and beyond what the user wrote needs to be thought about quite carefully.