Is there any documentation on what all the sections in hyperfine's output means?

[hwittenborn]

What exactly do all of these symbols in Hyperfine's output mean? (the ±, [User: 1.6 ms, System: 1.5 ms] part, etc etc.)

I tried looking at the man page for Hyperfine, but I didn't get anywhere with it.

[sharkdp]

Thank you very much for your feedback. I agree that this should be documented.

For now:

• As the (mean ± σ) string at the start of the line indicates, the green time in front of the ± sign is the mean time of all benchmark runs. The number after the ± indicates the magnitude of the statistical error on that mean value. It's the (corrected sample) standard deviation of all measurement times.
• The "User" and "System" times are also averages across all benchmark runs. "User" corresponds to the time that the process spent in user mode while "System" corresponds to the time that the process spent in kernel/system mode. You can find more information here: https://stackoverflow.com/a/556411

[hwittenborn]

All that made sense except the standard deviation stuff, even with that link you provided.

Sorry to have to ask, but would you have any dumbed-down version of how that works (more specifically the corrected sample aspect, the meaning of a standard deviation itself makes much sense). If the only real way to explain it would be to try to go through the math equation though it'd probably be better to just leave it there for people to look into though.

[sharkdp]

Assume you have a certain program (prog). And hyperfine prog tells you that it takes mean ± σ to run that program. Then you can assume the following. If you run prog ONCE, you will have a 68% chance that the runtime will be within the interval [mean - σ, mean + σ]. And there is a 95% chance that the runtime is within the interval [mean - 2σ, mean + 2σ]. For details, see [1]. So σ is a measure for the typical spread of the runtime around the mean.

Because hyperfine can only run a restricted amount of runs (N_runs), we can never know the true distribution of runtimes. But we can get an estimate for the true standard deviation by computing the corrected sample standard deviation from the list of runtimes that we have measured. If we do not use the corrected sample standard deviation*, we will underestimate the true standard deviation.

Behind all of this is the assumption that the runtimes are normally distributed. In reality, this is typically not the case. Runtimes often follow asymmetric, long-tailed distributions. Nevertheless, the estimated standard deviation is still a good measure of the typical spread around the mean value. If you want to dive deeper into the statistical analysis of benchmark results, you can use the --export-json option and then run the advanced_statistics.py script from https://github.com/sharkdp/hyperfine/tree/master/scripts.

* The uncorrected sample standard deviation has N_runs in the denominator, the corrected version has (N_runs-1) in the denominator.
[1] https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

[sharkdp]

@hwittenborn does that make sense?

[hwittenborn]

Sorry that's my bad, I was still a bit confused on some things and haven't been able to look into it all recently.

The only thing I'm still confused on is the standard deviation as was the case before, I just hadn't felt like I'd looked into enough to ask for help again or anything yet. I'm still trying to look into it, there's just been a lot of stuff like working on some projects and school that's been stopping me from actually taking some time to figure everything out.

I'd say I could figure it all out once I get a chance to work it all through, though I think you've already adequately explained what exactly the output means itself, I just need to understand the underlying concepts of that all. With that, I'll leave it up to you to close the issue, but I think you've already summed up what the output itself is representing.

[hwittenborn]

I just started learning about some of this stuff in my math class, and more of it's making sense anyway. I know how standard deviation works, though I'm not too familiar with how corrected samples work, though that's all stuff I can figure out on my own at this point.

After again looking at the output in the original issue post, it's all kind of self-explanatory considering the text that's used. This issue's good to be closed on my end.

[sharkdp]

Let's reopen this until we have actually documented it somewhere.