Finalize story for libtest

[alexcrichton]

One of the last remaining crates to have a stabilization story in the standard
distribution is the libtest library. This library currently provides the backing
infrastructure and test harness used when rustc generates a test execuable via
the --test command line flag.

It's well known that libtest is not the end-all-be-all of testing frameworks. It
is intentionally minimal and is currently quite conservative in its scope of
what it tries to accomplish as well as what it implements. A testament to this
is the fact that one very rarely writes extern crate test, and it almost means
that the stabilization story need not be considered for the crate at all! The
benchmarking feature of the compiler, however, is quite useful and is one of the
sole reasons for using extern crate test.

When benchmarking, there are a few primary interfaces to the libtest library
that are used:

• test::Bencher - the type for a benchmarking harness.
• test::Bencher::iter - a member function used to run a benchmark.
• test::black_box - a useful function to hinder optimizations and prevent a
  value from being optimized away.

These three pieces of information are the primary targets for the stabilization
in this commit. The rest of the testing infrastructure, while still quite useful
to some projects, is not in scope for stabilization at 1.0 and will remain
#[experimental] for now.

The benchmarking pieces have been moved to a new rustc_bench crate which will
be part of the standard distribution. In order to write a benchmark one will
need to import the crate via extern crate rustc_bench and otherwise all usage
remains the same. The purpose of this crate is to provide a clear area for these
benchmarking utilities as well as provide a clear name that it is only
intended for use via the compiler #[bench] attribute. The current interface is
quite minimal with only what's necessary as #[stable].

It is most certainly a desire for the compiler to support other testing
frameworks other than the standard libtest. This form of infrastructure (be it a
plugin or a separate interface) is out of scope for 1.0, however, and this
commit does not attempt to resolve it.

Due to the removal of benchmarking items from libtest, this is a breaking
change. To update, rewrite imports of extern crate test to extern crate rustc_bench and then rewrite all use statements as well. The public interface
of the Bencher struct has not changed.

[breaking-change]

[rust-highfive]

r? @nick29581

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

r? @aturon

cc @brson

[aturon]

LGTM. @brson are you comfortable with this strategy?

[brson]

@aturon I'm not sure yet.

[brson]

I don't know that I'm comfortable stabilizing and standardizing a crate called 'rustc_something'. This is a crate that will need to be implemented and supplied by all Rust implementations. Is the motivation here to save the name test for a better testing crate?

[brson]

My bigger concerns around stability of --test is the various ugly flags that the main method of the test crate accepts.

[brson]

If we're uncomfortable exposing a good crate name here we could instead inject a 'std test' prelude - this arguably seems better to me anyway since #[bench] is unusable without the imports.

[alexcrichton]

Yeah I was trying to hide the test crate entirely to avoid taking the name and to avoid committing ourselves to distributing that crate itself. I agree with you though that I'm still a little uncomfortable with this because we're still forced to distribute something. That and the "you need to import this always" is a little unfortunate.

I can think of a few alternatives:

1. Don't stabilize the benchmarking infrastructure entirely. This involves taking no action from where we are today.

2. Alter the signature of benchmarking functions to only require libstd types. For example each benchmarking function could have a signature like:

   • fn() -> Box<FnMut()> - the "run one iteration" closure is returned after some setup, closing over all necessary variables.
   • fn(f: FnOnce(&mut FnMut())) - the "run one iteration" closure is handed to f which then executes the entire benchmarking job.

   The downside for both of these is that we don't get the nice "MB/s" display as we can't set the num_bytes variable, but the upside is that you don't need to import anything and it should all "just work" without extern crate test. Another downside is that there's nowhere for black_box to go.

3. Stabilize the name test, but only have Bencher/black_box as #[stable] inside of libtest.

I'd almost personally lean towards option 1... thoughts @brson or @aturon?

[alexcrichton]

After some more discussion on IRC, I'm going to withdraw this and continue pursuing a strategy after the alpha release. This means that for the alpha release crates will need to opt-in to the unstable apis provided by the test crate, which is similar to our story for other APIs for alpha.