A handful of changes to improve human-readable output and offer better UX#326
Conversation
The `{summarize,effect_size}::write` functions now include each benchmark's
median in the bracketed statistics.
Prints, for example, "pulldown-cmark" instead of "benchmarks/pulldown-cmark/benchmark.wasm".
Add a shared write_table helper and use it from both `summarize::write` and
`effect_size::write`.
Also indent output in four-space steps to make it easier to visually parse.
Example output:
```
compilation
Sum Total
cycles
┌────────┬────────┬───────────┬────────┬─────────────────────┐
│ Min │ Max │ Mean │ Median │ Engine │
├────────┼────────┼───────────┼────────┼─────────────────────┤
│ 186145 │ 229974 │ 213469.60 │ 221862 │ /tmp/old_backend.so │
└────────┴────────┴───────────┴────────┴─────────────────────┘
```
And adds a `--color=auto|always|ansi|never` option to control this behavior.
Also adds a `--show-instantiation` flag to skip the filtering. Instantiation measurements are dropped before computing effect sizes unless either `--show-instantiation` or `--benchmark-phase instantiation` is passed.
`display_effect_sizes` now hides insignificant results, other than the "Sum Total" results, which are always retained. This greatly removes clutter from the output, focusing attention only on the results that matter. Note that this also adds a `--show-insignificant` flag to preserve full output, if desired. When any results are hidden due to insignificance, we print a message reporting how many were hidden: ``` 12 statistically insignificant result(s) hidden. Note: You can pass `--show-insignificant` to show statistically insignificant results. ```
Otherwise, always prefer reporting effect sizes instead, as long as we have at least two different engines. (Previously, we would only report effect sizes when given exactly two different engines). This allows for easy N-way comparisons between engines.
|
(Thanks for all the reviews lately @posborne; almost done here!) |
posborne
left a comment
posborne
left a comment
There was a problem hiding this comment.
This is great, thanks for making these changes @fitzgen. Definitely an improvement to usability.
I have considered some if we want to change the defaults further to default to capturing raw results and separating data collection from analysis even further. Given how long it can often take to capture the data for a benchmark run, not having the raw results captured seems unfortunate. In any case, these changes are good regardless and I believe the tooling for doing cli-based analysis continues to work using inputs from file.
No objections to landing; just tangential thoughts on UX for typical/optimal usage.
For sure, this is something I've thought about before as well. On the one hand, as you point out, it sucks to re-run benchmarks when you just want to analyze the results in a slightly different way. On the other hand, people want a single command that DWIM and "just works"; putting any friction on that path means they will just do a one-off test, probably with worse methodology and less representative benchmark programs. I've been thinking that a best-of-both-worlds solution could be to do something similar to what What do you think of this idea? |
2 participants
Here is an example of the new output: