A note of caution on putting benchmarks inside the crate

[dralley]

Rust doesn't inline across crate boundaries unless you use #[inline] or LTO, whereas inside the crate it can inline anything it thinks might be worthwhile. That means that a benchmark defined inside a crate may not be getting an accurate measure of the performance that a user of the crate would see.

A practical example of this was just noticed in a library I use where the existing (libtest, not Divan) benchmarks inside the crate showed nearly twice the performance as Criterion benchmarks outside of the crate.

Therefore, while putting benchmarks inside of your crate is very convenient, it is not necessarily a good idea in all cases.

shepmaster/jetscii#57 (comment)