ASV build matricies

[hmaarrfk]

I think the benchmarking suite is very verbose.

scikit-image/asv.conf.json

Line 58 in 335de98

"matplotlib": ["2.2"],

• I don't think we need six anymore, but maybe when we run historical benchmarks?
• I don't think we should pin matplotlib. Very little actually uses it.

Currently, the matrix will build 2 * 2 * 2 * 1 * 1 = 6 environments. This is a little excessive and takes a long time to build on local machines.

Is it possible to have 2 builds?

1. A "1 year old configuration": numpy 1.14 and scipy 1.0
2. A bleeding edge configuration. numpy 1.15 and scipy 1.1

Is this possible to configure?

pinging: @jni @TomAugspurger since they worked on getting ASV up and running
pinging: @stefanv since it seems benchmarks are being discussed in numpy as well

CC: @serge-sans-paille since he is working on pythran stuff that might benefit from this discussion

[stefanv]

Yes, I imagine slowdowns are usually across platforms, so two entries seem reasonable.

[grlee77]

closed as outdated. we only have 1 entry that gets run at the moment (Python 3.8, numpy 1.17)