Shorter output for `rustc --test` binaries.

[SimonSapin]

Until now, a program created with rustc --test prints at least one line per test. This can be very verbose, especially with data-driven tests when hundreds or thousands of tests is not rare.

This changes the default output to one character per test (except metrics and benchmarks results which have additional data to show):

     Running target/debug/wpt-75c594dc1e6e6187

running 314 tests
..............................................................................
..............................................................................
..............................................................................
..............................................................................
..
test result: ok. 314 passed; 0 failed; 0 ignored; 0 measured

The previous behavior is available by passing --verbose to the test program. Maybe cargo test --verbose could be changed to do that? Edit: the default is now unchanged, -q or --quiet enables the new output.

[alexcrichton]

I personally think that the ship has unfortunately sailed on changing libtest too much right now. We've talked a bit about custom test frameworks/runners recently and this would certainly be easily fixed by a custom test runner, and until that time we may just not be able to change the format in terms of stability

In terms of personal opinion, I can see the value in having a more terse output but I prefer to see the longer output. It allows easily diffing two runs to see which test deadlocked, in single threaded mode it shows which test is running (to see what deadlocked), etc.

[SimonSapin]

Regarding stability: would it be better if this new output was not the default? Maybe with a .cargo/config pref.

Regarding preference: that’s why the old output is still available with --verbose. (Single-threaded mode is not the default either, so you have to opt into something anyway.) Sure, when things go wrong more output can help diagnose what’s going on. But most of the time flooding the terminal with hundreds of lines of test foo... ok is just noise that hides relevant bits of outputs like build warnings.

[alexcrichton]

Oh last time I meant to cc @rust-lang/tools

I don't think .cargo/config has much to do here unless we want to add a TOML parser to libtest, and I'd also be more ok with adding a flag to get output different than what's there today

[SimonSapin]

Re .cargo/config you’re right, I was confused with cargo test.

For what it’s worth, I’ve applied this to https://crates.io/crates/rustc-test. You can see it in action in https://travis-ci.org/servo/html5ever/jobs/111813368#L281

[alexcrichton]

Ok, we discussed this in the tools triage meeting today, and the conclusion was that we probably want to leave the output the same as-is today, but we're fine with a flag being added to enable this output. Would you be ok with that @SimonSapin?

[SimonSapin]

Alright. I’ve amended the commit to flip the default.

[alexcrichton]

This and the previous test can be reverted to what they once were, right?

[alexcrichton]

Thanks! Can you also add a test for this switch + output?

[SimonSapin]

Done.

[alexcrichton]

@bors: r+ d23fd71

[bors]

⌛️ Testing commit d23fd71 with merge 74dfc1d...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt, auto-win-msvc-64-opt-rustbuild