[FR] Benchmark series

[gustaphe]

Idea: Some way to specify the name of a special size parameter in a Benchmark, so (something like)

b = @benchmarkable bogosort(x) setup=(x=rand(n)) size=:n
res = run(b, 10)

would time bogosort with a size n=10. With this, a type BenchmarkSeries which encapsulates a Benchmark or BenchmarkGroup and a vector of sizes could let you investigate the size dependence of a benchmark:

function Base.run(bs::BenchmarkSeries{Benchmark})
    map(bs.sizes) do size
        time(run(bs.benchmarkable, size))
    end
end

This would help you answer questions like "how large should a vector be before algorithm A overtakes algorithm B?".

[vchuravy]

That sounds like a nice addition!

[gdalle]

How different is it from creating a BenchmarkGroup with various values of n as keys?

[gustaphe]

Functionally similar, interface wise quite different. Say you're comparing four different algorithms, and you want to benchmark them for size 10, 100, 1000, 10_000, etc. Handling this with a benchmarkgroup becomes unwieldy, you can of course loop through and create dynamic keys, but that's a lot of boilerplate. And the display of your result will not automatically respect the inherent structure. A benchmarkseries could have a plot recipe connected to it, a numbered benchmarkgroup not really.

[gdalle]

I think this could be achieved by #256 without changing the interface?