Tracking issue for #[bench] and benchmarking support

[alexcrichton]

This is a tracking issue for the #[bench] attribute and its stability in the compiler. Currently it is not possible to use this from stable Rust as it requires extern crate test which is itself not stable.

Core APIs for benchmarking:

• #[bench], which means the function should take a &mut Bencher argument and will be run as part of regular tests (and specifically benchmarked with --bench passed to the test binary).

crate test {
    mod bench {
        #[derive(Clone)]
        struct Bencher { ... }

        impl Bencher {
            fn iter<T, F>(&mut self, inner: F)
            where
                F: FnMut() -> T;
        }       
    }
}

[alexcrichton]

Nominating for discussion in 1.6, and cc https://internals.rust-lang.org/t/bench-status/2122/10, a thread about this.

I've currently extracted libtest and I would personally like to deprecate #[bench] with a little bit of extra support from perhaps Cargo or the compiler. We'd basically be saying "let's have benchmark support develop externally first" before moving it back in to the compiler.

[nagisa]

I like bench being as easy to use as it currently is, but would like more functionality.

IMO it would be nice to be able to provide custom providers for --test bineries so you could compile rustc … --test and run binary --bench, binary --quickcheck, binary --criterion etc for whatever testing and benching libraries you’re using.

[alexcrichton]

Yeah my feeling here is that I wouldn't want to stabilize anything in the compiler itself unless it looks like a generic "custom test framework" infrastructure, but that's unfortunately a big undertaking that may take awhile. As a result my personal preference is to deprecate the support in the meantime for technically-the-same-if-not-quite-as-easy-to-use support on crates.io

[alexcrichton]

The libs team decided to punt on this for 1.6

[briansmith]

This looks like it might be a good replacement for the current #[bench] mechanism: https://github.com/japaric/criterion.rs.

I agree with Alex that deprecation is the best resolution.

[Aatch]

This just came up on IRC and should be a consideration for anything we do here. Allowing for "unbenchmarked" parts of an iteration. The specific case this came up for was benchmarking an in-place sorting algorithm. Since performance is going to be different for sorted vs. unsorted lists, you need to clone the list each iteration which can throw off the results.