Clarify whether binary crates and private functions can be benchmarked

[Seeker14491]

I have a binary crate that has some functions I want to benchmark, but upon trying to use Criterion, it seems to me that binary crates are not supported, as I can't call into functions that are in a binary crate. Additionally, it's unclear how to benchmark functions that are not part of a crate's public API.

[Eh2406]

I think the part of the docs to be clarified is "This means that it's not currently possible to include benchmarks in the src/ directory."

[bheisler]

Hey, thanks for trying Criterion.rs.

As far as I'm aware, it should be possible to call public functions in a binary crate in your benchmarks. I can't think of anything that would prevent that. Could you share more details?

As for benchmarking non-public functions, this is not possible. Cargo does not expose any way for Criterion.rs to define benchmarks in the main crate (ie. in the src/ directory). Each benchmark (as defined in Cargo.toml) is actually compiled as a separate crate, and so it can only access things which are visible to outside crates. Unfortunately, there is currently nothing I can do about this; it would require some changes to Cargo and rustc.

[Seeker14491]

Regarding binary crates: writing extern crate <the binary crate> in the bench .rs file produces this error:

error[E0463]: can't find crate for `mybin`
 --> benches\my_benchmark.rs:4:1
  |
4 | extern crate mybin;
  | ^^^^^^^^^^^^^^^^^^^ can't find crate

Therefore I cannot reference anything in the mybin crate.

Regarding benchmarking non-public functions, might it be possible after the Custom test frameworks RFC is implemented?

[bheisler]

It looks like no crates can depend on a binary crate, so external tests and benchmarks don't work - see rust-lang/cargo#4316 for more details.

I've expanded and reworded the Known Limitations section in the User Guide to explain this in more detail. Thanks for the suggestion!