Continuous benchmarking with ASV

[alimanfoo]

It would be great to implement ASV or similar so we catch any changes that introduce a performance regression.

[jakirkham]

@joshmoore this was something we were looking at right?

[joshmoore]

@jakirkham, yes, during the last GSOC round. There are a few decisions to be made and then various functions to be written, but pretty straight-forward.

[jeromekelleher]

Just FYI, it's worth having a look at upstream discussions about the future of the project (e.g. airspeed-velocity/asv#1219 ) before committing to ASV

[joshmoore]

Oh wow. Thanks for the heads up, @jeromekelleher! (Yet another reason it was sad to have missed the summit.) I'll keep an eye on it, but in our testing with asv so far (for zarr-python) there was already one "speed regression" identified. (By another "j" programmer who shall rename nameless.) So it might make sense to get something in place soonish, with the understanding that we will be up for jumping ship when the time comes.

[JackKelly]

Just following up on this: Were any benchmarks implemented for numcodecs? If so, did the benchmarks use ASV?

it's worth having a look at upstream discussions about the future of the project (e.g. airspeed-velocity/asv#1219 ) before committing to ASV

For context: I'm currently researching benchmarking tools for benchmarking zarr-python. ASV looked good. Until I saw that discussion!

[joshmoore]

In the context of GSOC asv was only implemented for zarr-python and there only superficially.