[benchmark] More Robust Benchmark_Driver

[palimondo]

Robust statistics are statistics with good performance for data drawn from a wide range of probability distributions, especially for distributions that are not normal.

This PR improves the benchmark measurement methodology used by Benchmark_Driver, as originally outlined in Towards Robust Performance Measurement. Building on the Benchmark_O's quantile report format from PR #19328, it now collects representative sub-sample (exact measurements up to sample size of 21, ventiles plus min and max for larger sample sizes) from benchmark measurements, which is filtered to exclude outliers (values higher than Q3 + 1.5 * IQR) and combined into one empirical distribution by merging results from -i independent samples. The long form of this argument is renamed from misleading --iterations to --independent-samples. The log output format is changed to report Five-number summary, replacing mean and standard deviation (not applicable, because the underlying distribution is non-normal) by quartiles.

To facilitate this change, the Benchmark_O was modified to use the same auto-scaling mechanism, previously used to determine number of iterations, to collect variable number of samples based on the --sample-time argument, when --num-iters=1. This type of measurement, without averaging of multiple values into the reported runtimes, is now used by the Benchmark_Driver. It now also reports time to run the setUpFunction in the --verbose mode.

Additional changes required to enable the above:

• LogParser support for parsing:
  • benchmark report summary in quantile format,
  • setup function timing, and yields from the --verbose output of Benchmark_O.
• BenchmarkDoctor validation:
  • new rule to enforce sensible runtimes for setUpFunction,
  • modified recommended runtime to be <1ms, runtimes >10ms are reported as errors.

---

I've structured the PR as a series of small, incremental changes with detailed descriptions in the commit messages, it is therefore best reviewed sequentially by individual commits.

[palimondo]

@eeckstein Please review. Please also run full test, to exercise the benchmark tests (if it's not part of the smoke-test yet...).

[palimondo]

I'm working on another PR, that should land in sync with this one, since this changes the measurement methodology used in Benchmark_Driver to always measure with --num-iters=1 and the benchmarks that exhibit setup overhead would report different minimum value, because the overhead is no longer amortized over multiple iterations.

[eeckstein]

@swift-ci benchmark

[swift-ci]

Build comment file:

No performance and code size changes

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

---

[eeckstein]

@swift-ci smoke test

[palimondo]

@eeckstein I've pushed the fix for linter nitpicks that stopped the last build. Please re-test. (Are benchmark tests now part of smoke test, too — or should we run full test as we did before?)

[eeckstein]

@swift-ci smoke test

[eeckstein]

@swift-ci smoke test