Rust Benchmark Rule

[UebelAndre]

There's currently a rust_benchmark rule as part of the core package for rules_rust. I'd revived a PR to fix it up (#923) but I'm not actually sure what a benchmarking rule would do. Is there prior-art for this in Bazel? I'd love to hear about existing or potential use cases.

[hlopko]

I'm wondering what is the value rust_benchmark rule adds here. I don't think passing benchmark arguments through rule attributes is all that generally useful, benchmarks often run in special environments, at special times, with different configurations. The benchmarking infrastructure traces the excution and collects data, draws plots and generates flamegraphs. Bazel will not be able to be customized to support all of that.

I'd vote for removing the rule and instead only use rust_binary for these use cases.

[UebelAndre]

By the will of the people, so it shall be done: #938

Though, I'm still interested in maybe a note or blog of sorts that can guide folks in how to setup criterion. Perhaps this discussion will stand as that?

[dfreese]

• As you said there is no prior art here for benchmark rules at $DAYJOB) we do xx_binary with tags=["benchmark"] but thats not anything that users really deal with.

I may steal that for my team.

[UebelAndre]

Status: It's been removed. Thanks for the review @dfreese

[hlopko]

Thank you for such a quick cleanup!

[GregBowyer]

dfreese you might want to include a "sh_binary" (which we did) or a "rust_binary" that does the fake out I pointed to.

Otherwise the benchmark just "tests" the code without being a benchmark :P

[sayrer]

Hmm, wouldn't it make sense in a cargo-raze context? I was actually planning to use this, assuming its behavior matches cargo --bench

[UebelAndre]

It does and it doesn't. I think cargo-raze can still be made to generate benchmark targets but would need to learn more about the available toolchains to ensure either bench was supported or otherwise would have a target that defines a dependency graph with criterion such that you'd just need to build the rust_binary. I think if libtest is ever stabilized then there will be a revival of rust_benchmark. I think at this point it's up to someone to advocate for the rule and open a PR kinda outlining some definitive goals the rule meets.

But that's just my own thoughts. I'd still like to see a benchmarking rule and have it play well with Bazel.

[briansmith]

When using Cargo, cargo test --benches and cargo test --all-targets will build all the benchmarks and run one iteration. We use this in our CI/CD to make sure that code changes don't break the benchmarks. We'd like this to keep working if/when we switch to Bazel.

Further, I agree that benchmarks ideally run in a controlled environment, and that might not be easy to do within Bazel. However, I'd like to be able to at least build the benchmark executables so that they can be transported to the controlled environment and run there. E.g. transfered to a physical Android reference device and run.

Also, in practice people get pretty good results running benchmarks on their local systems, and it would be very nice to be able to keep doing this, if we were to switch from Cargo to Bazel.

[hlopko]

Good points! Bazel definitely has to be able to build benchmarks, and it also has to be able to run them. Both rust_benchmark and rust_binary satisfy both these requirements.

The question I have is if we need a custom rust_benchmark rule that behaves exactly like rust_binary, just implicitly adds a dependency on the benchmark library, and gives you a way to hardcode number of iterations or other relevant configuration in the BUILD file. I suspect that people want to use multiple benchmark libraries, and they want to configure the benchmark differently. I worry that this benchmarking rule would end up looking like a gallery of different setups of different people and companies.

In my experience using rust_binary, explicitly depending on the benchmark library of choice, and storing the number of iterations and other relevant info in the CI configuration or somewhere externally to the build, is better at satisfying varying requirements of users, at a very little convenience cost.

I'm happy to discuss concrete use cases in detail, for example @briansmith mentioned they need to be able to build all benchmarks. Is it an option to communicate this information to the benchmarking library via an environment variable that would be set during the CI build?