#[bench] attribute is stable and test crate can be replaced

[brson]

Per @SimonSapin's comment. It is possible to swap out the test crate via cargo and get the #[bench] functionality on stable.

This has been abused in the wild for a long time, and is a fairly major blunder. Swapping out the test crate exposes a large swath of ad-hoc test API's to the stable ecosystem (rustc generates calls to these APIs).

Fixing this will require providing a plausible alternative and a long deprecation and migration period.

[SimonSapin]

As I said in that comment, I found zero evidence that anyone else used benchmarks this way or even knew it was possible. None of rustc-test’s reverse dependencies use #[bench]. (They use the API to dynamically generate tests.) I myself only did it in an empty temporary crate to check if it was possible. Do a crater run to be more confident, but I believe nothing will be affected if the #[bench] attribute is made unstable right now. (Gated on the test feature which is already enabled by crates that use it on Nightly without the dependency shadowing trick.)

As to exposing test API’s, that’s only API of the rustc-test crate (or other similar crate) which is a copy, separate from Rust’s original src/libtest. The former won’t be affected automatically if changes are are made to the latter. And if porting a breaking change is desired, it can be made in a SemVer-breaking version number, as usual for anything on crates.io.

[DrRibosome]

Hmm, ive been using this to get bench on stable for a bit now (but I dont have crates pushed to crates.io). It would be annoying for sure if this broke given that benching is useful, but I want to work on stable

[SimonSapin]

Since my previous comment, std made a breaking change to the unstable private data structures that rustc-test relies on: servo/rustc-test#4. rustc-test working on "stable" Rust is an illusion: it’s really using a loophole in the stability system to access unstable features.