Allow benchmarking of sequential functions

[jni]

From my reading of Writing benchmarks, setup functions are run once per benchmark. Sometimes, however, one benchmark's test is another benchmark's setup. When functions take a long time to run, it'd be great to be able to use the results of a benchmark as the setup for the next one.

Here's my own use case:

• load data
• build training graph (needs time and memory benchmarking)
• run one training epoch (time + memory benchmarking)
• run four more training epochs (they depend on the first and are qualitatively different) (time + memory benchmarking)
• build testing graph
• build testing priority queue (needs time and memory benchmarking)
• process testing graph (time + memory benchmarking)

Most of these are big operations that take some time to run, so it's annoying to have to run the whole pipeline again to run the final benchmark. And the objects are big and complex, so pickling saves some time but is still expensive.

Any suggestions?

Thanks!

[pv]

This probably goes to the feature wishlist, not directly supported right now. Some overlap probably with #179 (the benchmark is logically a single benchmark with multiple returns, if the different parts cannot really be run separately).

You can sort of cobble it together by running all of the steps in setup_cache, doing the measurements using your own custom memory and time tracking, and saving the results to cache. The cached results are then reported to asv by a bunch of track_* "benchmarks" that just return the cached benchmark result values. This still allows you to benefit from the reporting and analysis parts of asv.

[jni]

@pv thanks! I'm always happy to hear that I didn't miss some completely obvious part of the docs that solves my problem. ;)

I will try your hacky suggestion. Incidentally, if you have any pointers about where to start implementing this properly in asv, I might have a crack at it!

[pv]

All benchmark running is done in `benchmarks.py` (manager process) and `benchmark.py` (benchmark runner/discovery process).

[mrocklin]

cc @mike-wendt you might appreciate the hacky suggestion here