_collect_cc_coverage script can silently fail to produce coverage

[UebelAndre]

Description of the bug:

I noticed an issue where rules_rust rules were failing to produce coverage reports when running with --experimental_split_coverage_postprocessing and found it to be caused by runfiles not being available to the llvm-cov call which expects test binaries to be available:
https://github.com/bazelbuild/bazel/blob/6.2.1/tools/test/collect_cc_coverage.sh#L95-L97

The script would fail with a File not found error and no coverage would be reported.

This to me seems like a miss. I can see a test not having any coverage to report if maybe there's no dependencies and --instrument_test_target is not set, but otherwise I fully expected bazel coverage invocations to fail if a genuine error occurs.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Run bazel coverage //... --experimental_split_coverage_postprocessing in rules_rust v0.22.0 on Bazel v6.2.1.

Which operating system are you running Bazel on?

Linux, MacOS

What is the output of bazel info release?

release 6.2.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

https://bazelbuild.slack.com/archives/CA31HN1T3/p1686230133034699 is the thread where this issue was triaged.

Any other information, logs, or outputs that you want to share?

No response

[UebelAndre]

#18634 (comment) seems like it would fix the issue of rules_rust not supporting --experimental_split_coverage_postprocessing.

[comius]

@c-mita please triage

[keith]

I started trying to make that script fail tests (I assume that would have made it more obvious in this case) #15420 but that breaks tests since they're currently relying on that because the host machines I guess aren't setup for coverage

[UebelAndre]

Running into this again. Would it be possible to have whatever tests that depend on this use an environment variable or something to toggle the silencing behavior? It'd be extremely valuable to be able to see when generating coverage failed.

[keith]

Good idea, I added a flag to support that and enabled it for bazel CI, I wouldn't think users should ever use this #15420

[keith]

alternatively we could hijack that env var to just set it to true or something, but this is probably more clear

[UebelAndre]

IMO the script needs to fail if any problem is encountered. Only the tests you mentioned in #18650 (comment) should use this flag and everything else should fail fast and loud so rules maintainers can start making sure coverage actually works.