Test case timing information and ordering

[mgeisler]

I think it would be nice if cargo test would output timing information about each test it runs. That way users can see where to spend effort if one tries to make a test suite run faster.

It would also be nice if one could influence the order in which test cases are started. Most of my tests finish in an instant, but I have one that does more exhaustive testing. It would be good to have that test started first so it can crunch away in its own thread while the other tests are processed on the other cores. Put differently: I don't want this test to be started last since I'll run out of quick and fast tests to run on my other cores.

Testing... Okay, it seems that Cargo works in the following way:

• Test functions are started in alphabetical order
• The --test-threads flag is set to the number of CPUs

The documentation says that

By default, all tests are run in parallel. This can be altered with the --test-threads flag or the RUST_TEST_THREADS environment variable when running tests (set it to 1).

I first read that as saying "we spawn a thread per test so they all start executing in parallel". That would alleviate my problem above since the overall time for the test run would become the time for the longest running test. However, testing reveals that --test-threads is set equal to the number of CPUs (which is more sensible overall, though twice the number of CPUs could also be a good default in case the tests do even a slight amount of I/O).

If Cargo had timing information from past test runs, then it could use that to start the slowest tests first. Assuming tests are independent, it seems to me that such an order should reduce the overall test time to the time of the longest running test. I think the timing information could be saved inside the target/ directory.

Does this sound like a good idea?

[alexcrichton]

Thanks for the report! I think this may be a better issue for rust-lang/rust? The libtest framework (the test harness) is up there

[mgeisler]

Yes, definitely. I started searching for the help text above and couldn't find it in Cargo :-)

I'll close it here and open a new issue in the right tracker.

[vitiral]

I couldn't find an issue on rust-lang/rust so I opened one here: rust-lang/rust#46610

Note that my issue only addresses the timing issue, not the ordering.