Divide by zero and other issues when detecting outliers

[tdulcet]

When calculating the modified Z-scores, if many of the times are the same, the MAD will be zero (see here), which means this line will divide by zero:

hyperfine/src/hyperfine/outlier_detection.rs

Line 33 in 7cdd1f2

xs.iter().map(|&x| (x - x_median) / mad).collect()

The above link has a simple example that you can add to your test suit to reproduce this issue.

---

Additionally, the test suite correctly takes the absolute value of the modified Z-scores:

hyperfine/src/hyperfine/outlier_detection.rs

Line 47 in 7cdd1f2

.filter(|&&s| s.abs() > OUTLIER_THRESHOLD)

However, the program does not:

hyperfine/src/hyperfine/benchmark.rs

Lines 397 to 400 in 84f3120

	let scores = modified_zscores(&times_real);
	if scores[0] > OUTLIER_THRESHOLD {
	warnings.push(Warnings::SlowInitialRun(times_real[0]));
	} else if scores.iter().any(|&s| s > OUTLIER_THRESHOLD) {

Which means that it is missing 50% of the outliers.

I found these issues while testing hyperfine and implementing my Bash port. Thank you for making this wonderful tool!

[sharkdp]

Thank you very much for your feedback and for this detailed bug report!

which means this line will divide by zero

I guess this could be related to #319 as well, but I did not take a closer look yet. Definitely looks like a problem.

[…] However, the program does not […] Which means that it is missing 50% of the outliers.

Oh that is bad. The comment of the modified_zscores function also seems wrong because it includes the absolute value ("whereas the modified Z-score is defined by |x_i - x_median|/MAD"), which would mean that everything is fine. But the implementation returns (x_i - x_median) / MAD.

How could this have gone unnoticed for so long :-/. The only comfort is that the outliers which were not detected are "fast outliers" - which are quite uncommon in practice.

[sharkdp]

Fixed in hyperfine v1.11.0.