Tracking issue for #[bench] and benchmarking support

[alexcrichton]

This is a tracking issue for the #[bench] attribute and its stability in the compiler. Currently it is not possible to use this from stable Rust as it requires extern crate test which is itself not stable.

[alexcrichton]

Nominating for discussion in 1.6, and cc https://internals.rust-lang.org/t/bench-status/2122/10, a thread about this.

I've currently extracted libtest and I would personally like to deprecate #[bench] with a little bit of extra support from perhaps Cargo or the compiler. We'd basically be saying "let's have benchmark support develop externally first" before moving it back in to the compiler.

[nagisa]

I like bench being as easy to use as it currently is, but would like more functionality.

IMO it would be nice to be able to provide custom providers for --test bineries so you could compile rustc … --test and run binary --bench, binary --quickcheck, binary --criterion etc for whatever testing and benching libraries you’re using.

[alexcrichton]

Yeah my feeling here is that I wouldn't want to stabilize anything in the compiler itself unless it looks like a generic "custom test framework" infrastructure, but that's unfortunately a big undertaking that may take awhile. As a result my personal preference is to deprecate the support in the meantime for technically-the-same-if-not-quite-as-easy-to-use support on crates.io

[alexcrichton]

The libs team decided to punt on this for 1.6

[briansmith]

This looks like it might be a good replacement for the current #[bench] mechanism: https://github.com/japaric/criterion.rs.

I agree with Alex that deprecation is the best resolution.

[Aatch]

This just came up on IRC and should be a consideration for anything we do here. Allowing for "unbenchmarked" parts of an iteration. The specific case this came up for was benchmarking an in-place sorting algorithm. Since performance is going to be different for sorted vs. unsorted lists, you need to clone the list each iteration which can throw off the results.

[fschutt]

I'm sorry, but this is still required in nightly 1.20. If I write:

#[bench]
fn bench_blur_image(b: &mut Bencher) {}

Rust complains that it can't find Bencher. You have to use #![feature(test)] and extern crate test still. Why is this not stabilized or that Bencher could be found automatically? Just asking.

[SimonSapin]

@fschutt That is correct, benchmarking with #[bench] is unstable on only available in Nightly at the moment. This issue is about tracking its stabilization.

[alkis]

What's the current status of this given the recent discussion here? Is there another ongoing discussion somewhere else?

[Manishearth]

rust-lang/rfcs#2287

[tomaka]

Latest development seems to be rust-lang/rfcs#2318

[steveklabnik]

Triage: no major movement recently