data interpretation: start emphasizing the meaning of best-case outcome (minimum duration)

[jgehrcke]

In #535 we started discussing the relevance of min vs. mean in performance analysis. There is a plethora of literature out there that I think elaborates rather well on the fundamental ideas, such as

• https://math.mit.edu/~edelman/publications/robust_benchmarking.pdf
• https://blog.kevmod.com/2016/06/10/benchmarking-minimum-vs-average/

In plotting, I think we should move away from showing only mean values, but show both: min and mean (neutrally expose both pieces of information).

In addition, I think we can safely emphasize min over mean (but that might be controversial). Most importantly, there is no one-size-fits-all solution, and we should do a great job explaining related concepts.

[jgehrcke]

Made good progress towards enabling that systematically via the introduction of the 'single value summary' concept in #1172 and #1183.

[jgehrcke]

https://conbench.ursa.dev/compare/benchmark-results/06511cc82d757d868000ce4783696dc4...06511ed95f977e968000accebf5ff277/ good example where a 'best of N repetitions' would yield a more useful signal than the mean value.

Given

[5.366551, 5.2435, 5.283533]

vs

[5.278719, 6.832488, 5.97085]

Given that very simplistic view I think it's pretty obvious that this alert for a potential performance regression is a false positive.

One could potentially emit a 'instability warning' instead because it seems like that the volatility extent is new.