Benchmark statistics

[pv]

Record more benchmark samples, and compute, display, and use statistics based on them.

• Change default settings to record more (but shorter) samples. Changes meaning of goal_time Makes determination of goal_time more accurate; however, no big changes to methodology --- there's room to improve here.
• Do warmup before benchmark --- this seems to matter also on CPython (although possibly not due to CPython itself but some OS/CPU effects)
• Display spread of measurements (median +/- half of interquartile range)
• Estimate confidence interval and use that to decide what is not significant in asv compare.
• Optionally, save samples to the json files.
• Switch to gzipped files.

The statistical confidence estimates are a somewhat tricky point, because timing samples usually have strong autocorrelation (multimodality, stepwise changes in location, etc.), which makes simple stuff often misleading. There's some alleviation for this currently there in that it tries to regress the timing sample time series looking for steps, and adds those to the CI. Not rigorous, but probably better than nothing.

The problem is that there's low-frequency noise in the measurement, so measuring for a couple of second does not give a good idea of the full distribution.

todo:

• more tests, e.g. printing and formatting is mostly untested
• do we really need to gzip?
• update documentation
• tuning for pypy

[coveralls]

Coverage increased (+0.7%) to 72.988% when pulling f953513 on pv:timing-estimator6 into cfe3d3e on spacetelescope:master.

[coveralls]

Coverage increased (+0.6%) to 72.971% when pulling bd4ddca on pv:timing-estimator6 into cfe3d3e on spacetelescope:master.

[coveralls]

Coverage increased (+0.6%) to 72.971% when pulling bd4ddca on pv:timing-estimator6 into cfe3d3e on spacetelescope:master.

[coveralls]

Coverage increased (+0.6%) to 72.941% when pulling 364e0ed on pv:timing-estimator6 into cfe3d3e on spacetelescope:master.

[wrwrwr]

Changing goal_time to mark a single repeat time is nice, but do I read it right that setting repeat = 5 will take 5 samples if number is fixed and 50 samples if it is left at zero? Why not make the number of samples equal to repeat in the adaptive case too (with a single setup per sample and a higher default)?

[pv]

Yes, I've got something like that in the works. This is partly orthogonal to this PR, however. . The benchmark methodology may still need tuning. One issue faced here not faced by perf is that the benchmarks that run later, run under different CPU/systemload/heating conditions than the benchmarks that run earlier. So the sampling probably should use multiple processes, and be done in A B C ... A' B' C' ... A'' B'' C'' ... order.

[wrwrwr]

Interleaving benchmarks would surely be good, comparability across separate import benchmarks was something I was aiming for. The whole samples/repeats logic would probably need to be moved up to realize that though (with a "sample" rather than a "benchmark" script?).

[coveralls]

Coverage increased (+0.7%) to 73.009% when pulling 8c41b73 on pv:timing-estimator6 into 390145a on spacetelescope:master.

[coveralls]

Coverage increased (+0.7%) to 72.991% when pulling 8c8fbe0 on pv:timing-estimator6 into 390145a on spacetelescope:master.

[coveralls]

Coverage increased (+0.9%) to 73.191% when pulling 2c5b78e on pv:timing-estimator6 into 390145a on spacetelescope:master.

[pv]

I think this is more or less finished now. Some fine-tuning may be useful to do later on.

The changes in benchmarking parameter default values may be annoying, but these should give better accuracy. The default runtime also becomes shorter. Big improvements on the accuracy probably will need stuff such as splitting the benchmark runs into several parts, to be run in an interleaved order to ensure long sampling time spans to capture the low-frequency noise properly. https://github.com/pv/asv/tree/many-proc

[coveralls]

Coverage increased (+0.9%) to 73.203% when pulling 26a42ba on pv:timing-estimator6 into 390145a on spacetelescope:master.

[coveralls]

Coverage increased (+0.9%) to 73.187% when pulling fc2045d on pv:timing-estimator6 into 390145a on spacetelescope:master.

[coveralls]

Coverage increased (+0.9%) to 73.233% when pulling 347bb53 on pv:timing-estimator6 into e581104 on spacetelescope:master.

[coveralls]

Coverage increased (+0.9%) to 73.253% when pulling 50ddeb8 on pv:timing-estimator6 into e581104 on spacetelescope:master.

[coveralls]

Coverage increased (+0.9%) to 73.268% when pulling 10fdd49 on pv:timing-estimator6 into e581104 on spacetelescope:master.

[pv]

After some dogfooding, I think this is ready to go.