Problems with Bazel's code coverage collection

[c-mita]

In this document, I have described what I believe to be the current short-comings in Bazel's support for code coverage collection.

These areas are:

• Insufficient documentation.
• Insufficient Starlark functionality for some rules (C++ and Rust).
• Confusing API (confusing flag names, mystic attributes, etc).
• Post-test processing being controlled by test rules; library rules have limited ability to influence coverage collection.
• Baseline coverage being unsupported.
• Linux being the only properly supported platform.

In the near future, I would like to propose a new coverage API for Bazel, assuming there is appetite for one and that it solves enough problems.

What I would like to know:

• Have I missed anything? Are there other issues that need to be addressed.
• Which problems are the most pressing?

[c-mita]

@fmeum, @ulfjack, @UebelAndre; you've all had interests in this area - any thoughts?

[fmeum]

I am pretty busy this week but will definitely provide comments and my personal ranking of the issues next week.

[c-mita]

Friendly ping ;)

[fmeum]

What I find important (most to least):

• Documentation on what works and what doesn't and how (LLVM, GCC, remote execution, merged reports, ...). Doesn't matter if this includes a bunch of experimental flags as long as interested users can learn about them in a central place.
• A list of rulesets and their level of support for coverage. Once we have an overview of what's missing, we can more easily determine which functionality is needed to improve the situation for rulesets.
• Some idea for how meaningful baseline coverage could be provided in a way that all rulesets can adopt. It would be interesting to see how baseline coverage works inside Google.
• macOS and Windows support.

What I find less important:

• Coverage formats other than LCOV. Focusing on a single format first seems more efficient and converting LCOV to any other format is a relatively simple post-processing step that doesn't have to be handled in Bazel.

[c-mita]

• Documentation for the builtin Bazel rules should be easy(ish). Including the external rules under bazelbuild should also be fine, although there is some bikeshedding discussions to be had as to where to describe current support.
• Baseline coverage does not seem technically difficult to add support for. The question is much more of "interface". I've been meaning to write a proposal (and prototype) but just haven't got around to it.

[jungleraptor]

Hello, we're using Bazel at Swift Navigation. Very excited to see there's renewed interest in this area as this has been a pain point for us, especially for C++.

I'd say that the most important things we would like to see are (in order of importance):

• Better (non-experimental) support for LLVM based coverage (C/C++ code specifically).
• Better support on Macos for C/C++ coverage
• More output formats than lcov (specifically we integrate with sonarcloud which currently requires us to use --coverage_report_generator to convert to their format).
• Previously we used gcovr as one of our coverage processing tools, which supported some nice features like being able to discard compiler generated branches from the final report (further reading). This makes a huge difference in useful branch coverage metrics. It would be great to have an equivalent in bazel.

[c-mita]

One of the motivations for me writing this (and for considering a new API) is Rust coverage integration. Rust needs to do very similar things to C++ and I really want to avoid layering hacks inside collect_coverage.sh and collect_cc_coverage.sh. Ideally Bazel would have no knowledge about how coverage works beyond handling LCOV dat files.

[tsr-boxbot]

Just chiming in to say Baseline coverage being unsupported. appears to be the only issue impacting the usefulness of the current state of coverage for me. My env is pretty similar to any other workplace using modern C++ and modern Linux.

[jab]

One issue we've had to patch: bazel coverage foo refuses to collect coverage for any of the tests in the foo target if only a single test in that target fails, due to this code. A failing test in foo should not defeat coverage collection for all the tests in that target.

I'm not sure if/when I can upstream my patch (assuming there'd be interest in accepting it), but it should be extremely easy for someone else to contribute in the meantime. Hope this is helpful.

[sluongng]

I think in general, there needs to be a more robust story for post-test artifact aggregation.

Until now, Bazel has been treating test actions as a special class of actions: produce a binary and run(execute) that binary to capture result at the leaf nodes of the graph. If you want to capture test outputs, you are advised to convert test actions into normal build action instead. Doing that would allow you to treat test results/output as build artifacts and create additional actions to depend on them. However, the tradeoffs are:

• You cannot register your tests into Bazel test result reporting system
• You have to manually instrument your test binary execution
• Migrating from existing rules setup could be expensive, especially native rules (cc, java etc...)

So I think a good step forward would be providing API/method/convention to allow other actions to depend on existing test execution actions. That way we could start building on top of it to provide coverage, profiling, and fuzzing support for tests.

[sluongng]

I think there are 2 ways of solving this:

• What i wrote above is one: build graph model enhancement
• The current Buck2 approach is another way: introduce a TestRunner API and delegate the data collection / aggregation to the Runner layer. The default Bazel test runner could stay as-is while others could plug their own runners into Bazel and perform the aggregation and collection, then make those data available to Bazel when needed.

The 2nd approach is a lot more restrictive vs what I proposed earlier, but perhaps, more approachable realistically.

[c-mita]

#19633 seems relevant but doesn't appear to be catching on.

[moroten]

I would be happy to continue on #19633 if there is a chance for it to be accepted. At the moment, Google does not won't to make Bazel as generic as that suggestion could make it. #19633 can potentially lead to multiple independent implementations of coverage, report generation etc.

I'll be happy to discuss things more at BazelCon next week and write up the conclusions afterwards.

[sluongng]

@moroten Do you have a quick conversation summary with Bazel team at BazelCon? What are the current expected actions/workarounds available?

[moroten]

#19633 opens up a generic API so that Bazel can do more than just building and terminate after running tests. This can lead to rule authors and the community reimplementing coverage, test reports, SBOM gathering into release packages.

Some voices inside Google would like to keep the API more restricted and let Bazel do what it can do today. Another opinion is that we cannot foresee where it may end up. What will the maintenance cost look like?

In my opinion, #19633 gives more freedom to the community to handle test and coverage reporting etc. in more innovative ways. We just need to keep the community on one track so that we can continue to share rules. One example is the linting convention that Aspect-Dev is forming with https://github.com/aspect-build/rules_lint