cargo test --tests also runs unit tests -- target filters are confusing

[RalfJung]

Problem

cargo test has flags --libs, --bins, --tests, --doc, which I thought (based on their documentation) would control whether the lib crate unit tests / bin crate unit tests / integration tests / doc tests, respectively, are being run. But that does not seem to be what happens, so either the documentation is wrong or it is very confusing.

Steps

1. Run cargo test --tests.
2. Notice how it runs not just the integration tests, but also the unit tests!
3. However, cargo test --bins runs only the binary crate unit tests.

This seems clearly inconsistent. I would expect cargo test --tests to be the same as cargo test --test a --test b (manually listing all tests), which is how cargo test --bins seems to behave.

Possible Solution(s)

No response

Notes

Generally the cargo test documentation for all these many flags that control which tests are being run, is unclear on whether the flag adds tests to the default test set, or remove the default test set to only run a small select set of tests. (This also affects other commands like cargo check.) For instance, cargo test --benches tests more than the default test set, but cargo test --bins tests less than the default test set, but the documentation for these two flags reads exactly the same. Some flags use the word 'only', which is maybe meant to indicate "reduce default test set", but --bins does not use that word and anyway this is not at all clear.

The behavior I would have expected is along the lines of "if any of these flags is present, then only what is specified by the flags is being run, no default test set", but that does not match implemented behavior. This then makes me wonder how I would even say "run only integration tests" (cargo test --tests doesn't do that), or "check only the benchmarks" (cargo check --benches doesn't do that).

Version

cargo 1.64.0-nightly (85b500cca 2022-07-24)
release: 1.64.0-nightly
commit-hash: 85b500ccad8cd0b63995fd94a03ddd4b83f7905b
commit-date: 2022-07-24
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Debian testing (bookworm) [64-bit]

[weihanglo]

Totally agree it is confusing. People usually invoke cargo test --help and see it say Test all tests. Then guessing like "I know cargo tests. Tests here must refer to all test targets". And it turns out to be all targets that has default test flag set.

It's hard to encourage people to use cargo help test, read all the lengthy doc¹, and figure out how it works. I believe changing the behaviour is a breaking change, so there indeed is an inconsistency between --tests and other target options such as --bins. I would suggest improve the short help message first and see how to expand the long version, though I currently have no thought about a better help message.

Footnotes

1. --tests
   Test all targets in test mode that have the test = true manifest flag set. By default this includes the library and binaries built as unittests, and integration tests. Be aware that this will also build any required dependencies, so the lib target may be built twice (once as a unittest, and once as a dependency for binaries, integration tests, etc.). Targets may be enabled or disabled by setting the test flag in the manifest settings for the target. ↩

[RalfJung]

So is there even a way to ask cargo to test (or check or ...) all test targets (the things that are by default in the tests/ directory, the things that can be selected via --test name) but nothing else? I expect this to be possible somehow, and there's no other flag which does this, so it must be --tests -- that was my reasoning.

[RalfJung]

cargo help test

Oh wow, I had no idea that this command even existed! Thanks for pointing it out, I will try to remember. :)

I now see it is mentioned at the end of cargo test --help. Not sure how I missed it. I guess I didn't expect there to be a short and long version of the help, so I didn't even go looking.

What's the reason why cargo test --help does not do the same thing as cargo help test?

[weihanglo]

So is there even a way to ask cargo to test (or check or ...) all test targets (the things that are by default in the tests/ directory, the things that can be selected via --test name) but nothing else?

Underneath the hod, integration tests and unit tests are handled by the same mechanism. They are all collected from functions with #[test] attribute by libtest. Here is an old write-up may help. #10361 (comment). If you do want do run all integration tests but not unit test in library target, try to leverage glob syntax such as cargo test --test '*'.

What's the reason why cargo test --help does not do the same thing as cargo help test?

I got a impression that @epage had mentioned it somewhere but can't remember.

[RalfJung]

If you do want do run all integration tests but not unit test in library target, try to leverage glob syntax such as cargo test --test '*'.

Ah, didn't know that was possible, thanks. I should really read those long docs in their entirety.

Still I feel like a flag is missing here, since we do have such a flag for all other similar cases, AFAIK.

[heisen-li]

The description of --tests in the cargo book:

By default this includes the library and binaries built as unittests, and integration tests.

But there seems to be a problem with this statement, the results of my tests using cargo test --bins to test cargo will not show up in cargo test --tests, implying that --tests doesn't test the binary unit tests? So it would make more sense for --tests to be modified to only accomplish integration tests and library unit tests?

Also, in cargo test -h, in order to maintain uniformity, should --bins and --bin be described similarly to --lib, modified to read:
--bin: Test only the specified binary's unit tests
--bins: Test all binaries' unit tests

[epage]

In case its relevant, #12915 made some minor wording tweaks in 1.75