Benchmarks of private code not running on macOS

[Fraser999]

I created a small repo to demonstrate the issue.

In the root, if I run cargo bench on Ubuntu, I see the expected benchmarking output, but on macOS, I don't.

[dvdplm]

This is what I see too. The "register anywhere" feature seems to be broken on macOS (I'm on Sequoia, 15.0 (24A335)).

[dvdplm]

@Fraser999 Did you by any chance dig deeper into this yet? Do you have a hunch about where the problem lies and/or which macOS version broke it?

[Fraser999]

Did you by any chance dig deeper into this yet?

No, I'm afraid I've not had time.

Do you have a hunch about where the problem lies and/or which macOS version broke it?

Also no. For reference, I'm on Sonoma (14.6.1). Potentially it has never worked on macOS, since the example here works for me and that's what's used in CI to test this functionality.

[dvdplm]

Yeah the example works for me too, always did. Never figured out how to do it outside the divan crate though.

It's such a killer feature, sad it's incomplete.

[jdidion]

The best solution I've found so far is to a) create a feature flag that you can use to conditionally compile benchmark modules, b) put the divan::main() call inside a main function in a benchmarks sub-module annotated with #[cfg(feature = "bench")], and c) call the crate::benchmarks::main function from the main function in the benches folder.

Note that you should only call divan::main() once. Calling it within the crate appears to register all functions anywhere in the crate, but calling it from code in the benches folder does not. I've adopted the practice of putting it in a sub-module of the top-level module (lib.rs or main.rs) in my crate.

So your cargo.toml looks like this:

[package]
name = "foo"
...

[dependencies]
divan = { version = "0.1.14", optional = true }

[features]
bench = ["dep:divan"]

[[bench]]
name = "benchmark"
harness = false

You have lib.rs:

mod bar;
mod baz;

#[cfg(feature = "bench")]
pub mod benchmarks {
    pub fn main() {
        divan::main();
    }

    // This is a good place to put any common code that is only
    // used in benchmarking
    pub fn my_bench_util() { ... }
}

You have bar.rs (baz.rs is similar):

fn my_private_fn() {
   ...
}

#[cfg(feature = "bench")]
mod benchmarks {
    use crate::benchmarks::my_bench_util;

    fn bench_my_private_fn() {
        my_bench_util();
        divan::black_box(super::my_private_fn());
    } 
}

And you have a benches/benchmark.rs file with the following:

fn main() {
    foo::benchmarks::main();
}

To run benchmarks: cargo bench --features bench

This isn't as nice as having the divan::main() call in benches but it's not too much worse - until #[cfg(bench)] stabilizes, you'll still need a feature flag to avoid compiling your benchmarks in release.

[nvzqz]

This is quite unfortunate, but thanks @jdidion for finding a decent workaround! I'll leave this issue open until this works as expected.

[dvdplm]

@jdidion I ran into the visibility crate the other day, which could be helpful to make your workaround less verbose.

@nvzqz I did some debugging on this issue; I didn't get super far, but I did manage to prove to myself that the benchmark does make it into the __mod_init_func section.
As I'm sure you are aware there has been plenty of discussion around the #[used] attribute and I was worried that we needed the new #[used(linker)] variant to prevent the compiler from optimizing out divan. This is not the case. There must be something else stopping macOS from actually calling the code before main, but I have not figured out why.