Skip to content

libtest filters should include the test crate's name in the string matched against the filter #141862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
RalfJung opened this issue Jun 1, 2025 · 4 comments
Labels
A-libtest Area: `#[test]` / the `test` library C-discussion Category: Discussion or questions that doesn't represent real issues. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. T-testing-devex Relevant to the testing devex team (testing DX), which will review and decide on the PR/issue.

Comments

@RalfJung
Copy link
Member

RalfJung commented Jun 1, 2025

Let's say I have a whole bunch of .rs files in my tests folder, like:

  • floats/lib.rs
  • time.rs
  • thread.rs
  • ...

Now I want to run just the float tests. Naturally I'll do something like cargo test -- float. Sadly that does not work since this runs the tests that have float as a substring in the name of the test (mod1::mod2::test_fn), but the name of the crate is not included.

This is particularly confusing in the rustc repo itself, where ./x test library/{core,std} -- float will run the float tests in libcore but not the one in libstd: libcore has a single large test crate with a module called floats, while std has a bunch of separate test crates, one of which is called floats.

I think libtest should be changed to put the crate name in the string used for filtering, thus avoiding this confusing and surprising behavior.

@rust-lang/libs-api I've been told the libtest CLI API is in your purview. Is there any chance the behavior of filters could be adjusted? This would lead to strictly more tests being included in any given filter, which at least won't silently reduce test coverage (except when combined with --skip) -- but it could theoretically break a setup where that then includes a test that does not actually work.

@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Jun 1, 2025
@jieyouxu jieyouxu added T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. A-libtest Area: `#[test]` / the `test` library C-discussion Category: Discussion or questions that doesn't represent real issues. T-testing-devex Relevant to the testing devex team (testing DX), which will review and decide on the PR/issue. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Jun 1, 2025
@tgross35
Copy link
Contributor

tgross35 commented Jun 1, 2025

I've also been bitten by this before, would be happy to see this change. Cc @rust-lang/testing-devex based on the Zulip discussion.

@epage
Copy link
Contributor

epage commented Jun 2, 2025

rust-lang/libs-api I've been told the libtest CLI API is in your purview. Is there any chance the behavior of filters could be adjusted?

@rust-lang/testing-devex is coordinating on efforts related to the long term direction of libtest, under the direction of libs-api.

I think libtest should be changed to put the crate name in the string used for filtering, thus avoiding this confusing and surprising behavior.

Currently, this is left up to the caller. For a cargo workflow, this would be cargo test --test floats.

I think libtest should be changed to put the crate name in the string used for filtering, thus avoiding this confusing and surprising behavior.

This would be a breaking change, particularly when --exact is passed in. One way around that would be a compatibility shim could be added where --exact matches both with and without the crate name. We'd have to track it separate than the test's identifier to make sure we didn't just turn --exact into a suffix match.

We'd also need to consider how this should be represented in json reporting. That isn't stable but we're working towards it.

We should also consider how this should work with --list which is stable.

As we clarify what the minimum expectations are for custom test harnesses, I feel like we'd need to maintain both behaviors because there is too much community expectations around the current behavior.

@epage
Copy link
Contributor

epage commented Jun 2, 2025

I've re-opened rust-lang/testing-devex-team#12

@weihanglo
Copy link
Member

I wonder if it is possible to put this kind of breaking change in the next edition, though having two filter logic might be more confusing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-libtest Area: `#[test]` / the `test` library C-discussion Category: Discussion or questions that doesn't represent real issues. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. T-testing-devex Relevant to the testing devex team (testing DX), which will review and decide on the PR/issue.
Projects
Status: No status
Development

No branches or pull requests

6 participants