Add --slowtest mode to test harness for finding the slowest test

[Manishearth]

Elixir has this. It would be pretty useful for speeding up large testsuites.

Probably should exist as a --slowtest N parameter to the test harness, which cargo test passes down.

Doubt this needs an RFC?

[LeopoldArkham]

Hey @Manishearth, I'd like to do this;

I have a few questions (new to the compiler):

• Is this only shown if all tests pass, or does it just ignore failing tests?
• Does this overwrite a regular test binary? Or produce a separate binary the way bench or build does?
• I see where in libtest I can get the metrics info and add the functionality. Would I need to propagate these changes elsewhere too?
• I assume we do this for unit and integration tests; do doc tests need it too?
• How would you recommend testing this?

(This is all assuming it doesn't in fact need an RFC)

Thanks!

[Manishearth]

Probably ignore failing tests. But only when all tests pass is also fine by me.

Does this overwrite a regular test binary?

We can build this instrumentation into the regular binary and have it take the --slowest N argument that turns on measurement and reporting. I think. Making it a part of the test-binary-build step instead is fine too.

I see where in libtest I can get the metrics info and add the functionality. Would I need to propagate these changes elsewhere too?

Eventually we'd need cargo changes. I'm not fully clear on how the testing stuff works to know exactly what you need to change; I understand there's libtest and also the codegen done for test binaries.

I assume we do this for unit and integration tests; do doc tests need it too?

Your choice, that can be added later.

How would you recommend testing this?

Unsure, perhaps ask in #rust-internals. We can also figure this out later.

(I'm not well right now so I can't dig up these answers for you unfortunately; I'm able to type but reading is hard)