Output format to JSON

[cecton]

I'm trying divan at the moment and find it super easy to get started with 🎉

Unfortunately I will need to integrate things into a CI. Normally I should find a way to get the results in a JSON file so I can create a comment on a PR that shows the results. (Example here)

Is it possible to do this with divan?

[cecton]

oh I just found it in the future plans, nvm

Divan has many opportunities for features. Future versions will have:
Better output options:
Machine-readable output like JSON and CSV.

[epage]

For myself, I'm wanting to integrate with bencher.dev which has a json format but also adapters, including for criterion, see https://bencher.dev/docs/explanation/adapters/

Also, as part of rust-lang/rfcs#3558, I want to look into if there is a standard structure we can use for across bench harnesses.

[nvzqz]

@epage I suspect that each benchmark harness is going to want to run things its own way and thus enforcing a single canonical format across the entire ecosystem might be a futile effort. So rather than enforce a certain schema on the ecosystem, it might be more productive/valuable to enable cargo/rustc hooks that help with information consolidation? Although perhaps this is rhetorical by kicking the can down to the API layer.

[alshdavid]

For me, Divan is by far the best benchmarking tool available in Rust as it allows for benchmarks to be placed in #[cfg(test)] blocks using #[bench] attributes so I really want to use it in all my projects.

Not having access to the benchmark data is currently the only thing preventing me from using Divan in my production projects as we typically run benchmarks in CI and automate adding a "benchmarks" comment to PRs that contain a breakdown, comparison, graphs, etc.

This is pretty easy to track without external tooling, just need the actual run data. I actually tried to parse Divan's stdout at one point 😆

Perhaps a simple unopinionated interim solution would be for divan::main() to return a DivanReport (or add a divan::main_with_report()). That way the output format could be managed by the consumer without prescribing an output format:

#[derive(Serialize)]
pub struct DivanResult {
  pub name: String,
  pub fastest: Duration,
  pub slowest: Duration,
  pub median: Duration,
  pub mean: Duration,
  pub samples: usize,
  pub iters: usize,
}

pub type DivanReport = Vec<DivanResult>;

From here I could obtain it after the run and store it as JSON, CSV, or even generate graphs from Rust directly

// bench.rs
#[cfg(test)]
mod foo_bench;
mod foo;

#[cfg(test)]
fn main() {
  let report = divan::main();
  println!("{}", serde_json::to_string(&report).unwrap());
}

[alshdavid]

Hi, can we get a progress update on this? 🙏

I'm looking to integrate Divan into my team's project and this is currently the biggest blocker