Support for named benchmarks

[philpep]

Related to #481

[coveralls]

Coverage increased (+0.3%) to 58.522% when pulling baff01d on philpep:benchmark-name into a14d8dc on spacetelescope:master.

[coveralls]

Coverage increased (+0.3%) to 58.522% when pulling baff01d on philpep:benchmark-name into a14d8dc on spacetelescope:master.

[coveralls]

Coverage increased (+0.3%) to 58.522% when pulling f1fe09b on philpep:benchmark-name into a14d8dc on spacetelescope:master.

[coveralls]

Coverage increased (+0.2%) to 58.49% when pulling 81c06eb on philpep:benchmark-name into a14d8dc on spacetelescope:master.

[pv]

I guess this is mostly OK. Comments:

• I think the attribute name should be the more specific benchmark_name instead of name
• The benchmark selection now needs to import the whole benchmark suite for each benchmark run, which may be expensive. Would it make sense in get_benchmark_from_name to first try importing the benchmark function based on its name, and only if that fails fall back to scrounging with disc_benchmarks?

[coveralls]

Coverage decreased (-0.09%) to 72.241% when pulling 56c3e2c on philpep:benchmark-name into cfe3d3e on spacetelescope:master.

[coveralls]

Coverage decreased (-0.09%) to 72.241% when pulling 3f14e67 on philpep:benchmark-name into cfe3d3e on spacetelescope:master.

[coveralls]

Coverage decreased (-0.09%) to 72.241% when pulling 3f14e67 on philpep:benchmark-name into cfe3d3e on spacetelescope:master.

[philpep]

Sorry for the delay.

Pushed new commits with benchmark_name attribute instead of name.

The current implementation of benchmark discovery by name use a generator, so it doesn't always import the whole suite. Nevertheless I re-implemented direct import of benchmark in 9fb848e
I'd prefer to write it shorter but didn't find a way. My opinion is that this is not worth to have this complexity since import of the benchmark suite should be insignificant comparing to the benchmark run. Let me know if you prefer to drop it and I'll update the PR.

Thanks !

[pv]

The benchmark suite import time can be similar to the benchmark runtime. For example, for the scipy benchmark suite, it's 0.5 sec vs. ~0.1 sec per-file. For pandas, it's ~ 1.0 sec total vs. 0.3 sec per-file (both with hot disk cache here).

[coveralls]

Coverage decreased (-0.05%) to 72.283% when pulling 9fb848e on philpep:benchmark-name into cfe3d3e on spacetelescope:master.

[pv]

Note that the 'benchmarks' package name should be replaced by dirname(root).

[philpep]

Good catch ! I updated the commit.

[coveralls]

Coverage decreased (-0.05%) to 72.283% when pulling fe0e2d3 on philpep:benchmark-name into cfe3d3e on spacetelescope:master.