Allow picking sort order of the output

[horenmar]

When I run hyperfine --parameter-scan version 0 9 './some-command-{version}, I get the results sorted by their speed. This can be often useful, which is why I believe it became the default, but I am specifically iterating over 10 different versions to get time series data, so I want the output in order.

I did this by asking for markdown output, which is in-order, but people also request that to be sorted by speed (#603, #601). Meanwhile, I want the default output sorted by command order...

I think the solution is to make the sort order customizable for all outputs.

[sharkdp]

Thank you for your request.

Adding a generic sort order argument that affects both the terminal output as well as the exported markdown (and other formats?) sounds like a good idea to me! We could potentially still use different default sort orders for the terminal and the markdown, but that might be confusing because with this new option, they could not be changed independently (?).

[horenmar]

I think that people wanting to have different sort orders per output type is going to be very rare unless you open up some sort of reporting extension API.

However, if you want to future proof things, you could do something similar to what I have in Catch2. Catch2 is a testing framework and allows multiple different reporters to be active, so I support both a generic --out parameter which sets the output reporters default to, but also reporter-specific options in reporter spec, like so --reporter console::out=-::colour=ansi --reporter xml::out=full-report.xml. By analogy, this would lead to something like --export-json bench.json::sort=speed.

(I picked :: as separator for Catch2 because other options were worse 🤷 e.g. : is ambiguous with full path on Windows, , is ambigous with list options, ; is Windows cmd command separator, ...)

[sharkdp]

I think that people wanting to have different sort orders per output type is going to be very rare unless you open up some sort of reporting extension API.

Yes, agreed. Let's just add a single global sort order option.

what I have in Catch2

Ohhh, you maintain Catch2. I love it 👍

[sharkdp]

This is now supported in https://github.com/sharkdp/hyperfine/releases/tag/v1.17.0

The output format changes completely if you use --sort command, but I did not find a good way to combine the two variants. See screenshots in #655. Feedback would be very much appreciated.