RFC: Add native code coverage support in Cargo #3287
Conversation
|
(Tarpaulin author) Making coverage easier to get for users on any system and reducing the overhead would definitely be a big quality of life improvement! Just a note on tarpaulin it has alpha support for the llvm instrumentation xd009642/tarpaulin#549, but it also plans in future of using probe-rs and providing options for embedded device coverage. Some way of reducing As well as these changes making the coverage more applicable to more domains, it would benefit crates like cargo-llvm-cov and tarpaulin and enable coverage in more specific environments. Plus making coverage work for 99% of users without needing a 3rd party tool would be a big UX improvement. |
ehuss
added
the
T-cargo
|
As a new user to rust I want to say that this in my opinion would be a great step in the right direction. It would help make rust adoption simpler as there would be a clear way to check code coverage without needing to check what creates provide this functionality and potentially need to deep dive into them to choose one. |
|
Bumping this. The Rust community has always heavily invested into its tooling. This is a feature found in many other ecosystems. It’d be great to see an ergonomic cargo flag as a solution. |
There was a problem hiding this comment.
You can actually specify which crates to test when giving the cargo test command in a workspace. I think this RFC should talk about this scenario right now instead of leaving it as a future possibility.
And IMO we should remove the "root crate" wording from the RFC because of the above reasons.
I'll add this in as opposed to leaving it as a future possibility. |
|
Good update. But I still see some references to "top-level crate" which is a bit confusing with the new changes. |
…e default crates instrumented via the `--coverage` command to the current RFC and remove from the future possibilities section.
I've removed the references to |
|
I'm very new to rust and not sure if I missed it somewhere higher up but wanted to give my opinion on back-end support from my humble point of view. Even if in the initial working version we only have LLVM support that is still better than no built-in support. As a new user for me it would clearly identify what is at least a generally considered a good way to do coverage testing without doing a lot of research on the topic. I got the impression this change was mostly targeted at beginners, and I think beginners often don't mind working with defaults to get results to get started. Like using LLVM or whatever other backend is required to get started. That is not to say it should be included in |
…sts in binary targets and examples.
|
@epage thank you for taking the time to review this PR, it is greatly appreciated. I've made the updates you requested and added doctests to the initial iteration of this RFC. |
There was a problem hiding this comment.
This RFC seems to propose two things: a generic mechanism for how tools (or in this case arguments) are applied to different crates in a workspace, and some specific API for running code coverage which uses that mechanism (unfortunately, the generic mechanism is only described in terms of the specialist use for code coverage :-) ).
I think that the generic mechanism would be useful for more than code coverage and adding support to it to Cargo in a way that it can be used with any flags or by any plugin would be super-useful (certainly some fun questions about the design though). I hope that such a mechanism could be used by coverage plugins to provide a better experience.
On the question of code coverage itself, I strongly agree it is something Cargo should help support, but I don't think it should get built in to Cargo. There are a lot of things a build tool could do (benchmarking, fuzz testing, linting, other static analysis, dynamic analysis (e.g., with MIRI), ...) and if Cargo does all of them it will become a confusing mess. It should be a platform for supporting all things, but only include default support for core functionality. I don't think code coverage makes that mark - some people like it, some people don't. There are many ways to do it using many different tools and I don't think that even if there is consensus that it is useful, there is consensus on the best way to do it. Therefore, supporting it in Cargo risks us either having multiple ways to do it (in the long term, which makes things even more confusing) or becoming outdated and we have to recommend people not to use the built-in support.
|
|
||
| ### Alternative 4: use existing custom subcommands to run code coverage analysis | ||
|
|
||
| Supporting this alternative would mean that there wouldn't need to be any changes to Cargo |
There was a problem hiding this comment.
Given that these options exist today, I think the tone of the RFC should be more about 'moving an existing plugin from the ecosystem to core Cargo' rather than 'adding a new feature to Cargo'. Concretely, that means justifying why such a move would benefit a large enough subset of users to justify the increased maintenance burden, and what the benefits of core support rather than ecosystem support actually are. Furthermore, if you're proposing something new rather than something which already exists then you need to justify why the things which exist (or at least their API) are less good than the proposal.
I think you should also consider an alternative where we add features to Cargo which enable the ecosystem plugins to be as good as a core feature, i.e., could we add features which let cargo-llvm-cov be as good as the proposal in this RFC?
|
Thank you for your response @nrc
I don't necessarily agree with this statement. Code coverage is an essential part of the development lifecycle. This only boldens the argument for writing tests and ensuring tests capture core functionality of any project. The Rust compiler also has a mechanism for gathering coverage data in two ways. Instrumentation based coverage is supported in stable and a gcc coverage implementation is in unstable. Having stable support for instrumentation-based coverage in |
So, personally, I strongly disagree with this. I think code coverage is a flaky heuristic masquerading as a metric and it encourages useless boilerplate testing at the expense of testing any edge cases which don't show up as branches. BUT, that isn't really important. I do think it is sometimes useful and I do think some people consider it an essential part of their development. However, my understanding of the wider dev community (which is what is important here) is that there is no consensus on it being essential and it isn't in the same tier as unit testing, benchmarking, or documentation. I think (and the Cargo team might have a different opinion) that to qualify for core support in Cargo, something must either have almost unanimous support in the community (like testing) or be something that we are specifically pushing as an advantage of Rust (e.g., Rustdoc). (or possibly be something which is important and fundamentally cannot be supported as a plugin). I don't think coverage meets these requirements, but perhaps it has become a lot more widely accepted recently? Anyway, I don't think being a plugin is a bad thing! Some really great tools are Cargo plugins rather than having built-in support (Rustfmt, Clippy, cargo audit, etc.). I would love for Rust to have best-in-class support for code coverage (as a plugin) and I'd support changes to Cargo to facilitate that.
It actually kinda worries me that there are already two implementations here. It seems like that means it is more likely for there to be more in the future. They are also closely tied to the compiler backend, so support with a different backend (e.g., Cranelift) or with a different compiler is likely to be different or non-existent. That seems like another good reason for this to be a plugin to Cargo rather than built-in. |
|
If I'm understanding plugins correctly then Tarpaulin is already a coverage plugin for cargo (among others) I found it in the list of plugins for cargo https://lib.rs/development-tools/cargo-plugins The only concern I have is that it comes back to being a bit confusing for new users which one they should pick |
Yes, Tarpaulin is a plugin along with llvm-cov.
I agree with your sentiment which is why I introduced this RFC as adding this support directly in Cargo to help alleviate some of those concerns. @nrc makes some good points around how plugins can be a good thing in terms of handling code coverage for Rust projects, but it is also clear that Cargo is lacking some features that would allow these plugins to have the full picture around which crates need to be instrumented. |
|
As per our discussion, I pushed out a separate RFC to support a generic mechanism for setting rustc flags via Cargo. This allows setting any Rust compiler flag, not just code coverage flags, and enables them only on the current crate being built. Dependencies including transitive dependencies will not have the flags applied. This new feature will make it easier to enable rustc flags regardless of the cargo command being run. |
|
To be honest, I see code coverage as being to Yes, it can be abused and misinterpreted, but so can lints, and both are a pain to implement and maintain without support from the compiler. |
|
@ssokolow I agree with this sentiment, thus the reason for opening this RFC. In my opinion having support for code coverage integrated directly into Cargo would be extremely helpful for a large group of Rust developers. I understand that there is some disagreement in how beneficial code coverage can be but having builtin stable support for it so those Rust developers that find it extremely essential in elevating code quality are able to easily use it. |
|
One quick note — I would want to make sure this knows to not clobber between coverage and non-coverage artifacts so that the last two calls in this sequence are no-ops: $ cargo test --no-run
$ cargo test --coverage --no-run
$ cargo test --no-run
$ cargo test --coverage --no-run |
That would not be the case. The |
|
I understand that that's the case if implemented directly as a way to modify rustflags. But I'd like to have that not be the case with |
|
Ahh I understand your comment now. That is very interesting, my only worry about this adding additional artifacts for each crate that is being instrumented and potentially having 'coverage artifacts' for each target being compiled which can quickly expand the number of artifacts for a simple build. This might be as simple as adding a separate target/out directory when the I do see the benefit of this approach and will certainly add it to the RFC as an Thanks for pointing out this alternative implementation :) |
|
Just posting an update: The Cargo team discussed this RFC a bit. I think in the short term we would like to see more experimentation done as third-party commands. We may consider more first-class support in the future, as it would make it more accessible. However, it seems like there is quite a bit of space to explore. It sounds like one of the biggest concerns expressed in this RFC with the current third-party approach (such as [profile.codecov]
inherits = "dev"
rustflags = ["-Cinstrument-coverage"]
[profile.codecov.package."*"]
rustflags = []I understand that is not ideal, and due to a limitation of the current implementation this can't be used via Another thing that would be helpful is to include more of a survey of code coverage in other ecosystems. What lessons can we learn from them? |
|
@ehuss Thanks for the detailed response! Yes, that hack should be sufficient enough to use as a workaround for the main issue of using third-party commands to handle collecting code coverage for a specific crate. My biggest concern is that this is not as ergonomic as I would have hoped for collecting code coverage as it adds more onus on the user of the third-party command to set these profile specific rustflags rather than making it ergonomic for the third-party command to handle this. I agree that having a better way of setting RUSTFLAGS on a crate-by-crate basis would alleviate this scenario and enhance how third-party commands collect coverage metrics and would make it simpler for a plugin to set the specific RUSTFLAG needed to collect coverage for a given crate. I have started this conversation here.
Could you elaborate on what you mean here? It would be great to get this documented to see the full scope of hurdles that would be blocking first-class support for handling code coverage within Cargo. |
| Cargo-llvm-cov currently leverages the `cargo-config` subcommand which is still unstable. To do so, | ||
| cargo-llvm-cov sets the `RUSTC_BOOTSTRAP` environment variable to allow its usage from a stable | ||
| toolchain. This is not a recommended approach especially for Rust developers that want to use | ||
| such tools in production code. |
There was a problem hiding this comment.
cargo-llvm-cov is now using cargo-config2 crate and is no longer depending on the unstable cargo-config subcommand.
This is not a recommended approach especially for Rust developers that want to use
such tools in production code.
Btw, if you think that development tools do this is a problem, you may want to open the issue to rust-analyzer that does similar things to the previous cargo-llvm-cov.
| of the generated `profraw` files? This is used to ensure each test run generates unique profiling data as opposed to overwriting | ||
| the previous run. | ||
|
|
||
| # Future possibilities |
There was a problem hiding this comment.
It might be worth mentioning here some kinds of tests that are supported by cargo-llvm-cov and do not seem to be supported by this RFC.
- other (builtin) subcommands (e.g.,
cargo run) - merge coverage of different runs
- coverage of C/C++ code linked to Rust lib/bin
cargo-llvm-cov also supports the following, but I think they are out of the scope of cargo.
- external tests
- third-party test libraries (e.g., trybuild)
- other (third-party) subcommands (e.g.,
cargo nextest)
|
I'm going to propose to postpone this RFC. The Cargo team agrees that we would like to see first-party support for coverage support and recognizes that the current options are not great. However, we have concerns about how this ties directly into the llvm tools, and the uncertainty on how that works with various scenarios (different LLVM versions, different codegen backends, different approaches of coverage instrumentation, etc.). Primarily, there isn't anyone on the team who has the capacity at this time to champion this feature. As a compromise, we may be interested in possibly seeing some unstable experimentation done in Cargo, where we can explore options for the solution. This would be with the intent that an RFC would follow up before stabilization. If anyone is interested in working on that, feel free to reach out to the Cargo team on #t-cargo on Zulip, or on the issue rust-lang/cargo#13040, or come to Office hours. We would probably like to see it broken into smaller pieces, such as adding an option to have Cargo pass the @rfcbot fcp postpone |
|
Team member @ehuss has proposed to postpone this. The next step is review by the rest of the tagged team members: No concerns currently listed. Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! See this document for info about what commands tagged team members can give me. |
rfcbot
added
proposed-final-comment-period
|
I think it would make sense to add a flag that lets you only generate the profraw files and stop there. This way you could manually merge the coverage of multiple executions of cargo test which you may be invoking for testing different configurations (different features, different targets, etc). |
rfcbot
added
final-comment-period
|
🔔 This is now entering its final comment period, as per the review above. 🔔 |
rfcbot
added
finished-final-comment-period
|
The final comment period, with a disposition to postpone, as per the review above, is now complete. As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed. This is now postponed. |
rfcbot
added
postponed
This RFC proposes integrating Rust's support for source-based code coverage into Cargo.
Internals Thread
Rendered
rust-lang/cargo#13040