The bencher should have a tool-friendly output format

[steveklabnik]

Issue by mzabaluev
Sunday Feb 22, 2015 at 19:17 GMT

For earlier discussion, see rust-lang/rust#22683

This issue was labelled with: in the Rust repository

---

The logfiles produced by bench runs have a format that is not very friendly to simple tools like join: the type of test/result and the test path do not have stable positions if the lines are broken up into space-separated fields.

It would be nice if the test driver could produce tool-friendly output where each test produces a line of regular space-separated fields with optional freeform information:

type path result freeform

Here, type can be test, bench, or ignored. path is the crate path of the test function. result is a single token depending on the type (time/iter for benchmarks, ok|failed|etc. for tests, not present for ignored tests).

[steveklabnik]

Comment by steveklabnik
Sunday Feb 22, 2015 at 19:40 GMT

---

We recently removed the JSON output of the benching tool, actually. And it's not to say that it's not a thing that's useful, but in an effort to stabilize, we're only committing to the minimum.

We're tracking feature requests for things like this over in the RFCs repo now, so I'm going to move it over there. Thanks!

[briansmith]

IMO, the output is tool-friendly enough. And, in particular, the output is more tool-friendly than JSON for many tools/languages, including Rust.

[Gankro]

How is the current format more tool-friendly for Rust, given that rustc-serialize and serde both exist?

[briansmith]

How is the current format more tool-friendly for Rust, given that rustc-serialize and serde both exist?

It is much easier to use a regex to parse the current output, than it is to even figure out how to get rustc-serialize or serde to parse JSON, particularly if you need to support stable Rust. Check out the README.md for serde: https://github.com/serde-rs/serde to see how complicated it is.

[huonw]

There's no requirement to take any of those routes, e.g.

let v = serde_json::from_str::<serde_json::Value>(&s).unwrap();

let number = v.find("foo").unwrap().as_i64().unwrap();
let string = v.find("bar").unwrap().as_string().unwrap();

(I imagine some judicious Index impls could help too, if serde_json wanted to take that route.)

Also, it would take just one library to define the right structs with the appropriate macro invocation/build script/deserializer to be able to reuse it, e.g.

extern crate bench_tools;
/* includes
struct Benches {
    benches: Vec<Bench>
}
struct Bench {
    name: String,
    ns_per_iter: u64
}
*/

let data = serde_json::from_str::<bench_tools::Benches>(&s).unwrap();

for bench in &data.benches {
     println!("{} took {}", bench.name, bench.ns_per_iter);
}

(Of course, the current line-based plain text format is nice for processing with shell scripts, but it's not great for more structured/"permanent" processing.)

[Gankro]

Oh, haha, I sometimes forget people endure the stable channel. 😅

(However I do take issue with the claim that regexes are "easy", and that today's output is particularly regex-friendly. In particular, large numbers get commas.)