Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Benchmark series #220

Open
gustaphe opened this issue Jun 5, 2021 · 4 comments
Open

[FR] Benchmark series #220

gustaphe opened this issue Jun 5, 2021 · 4 comments

Comments

@gustaphe
Copy link
Contributor

gustaphe commented Jun 5, 2021

Idea: Some way to specify the name of a special size parameter in a Benchmark, so (something like)

b = @benchmarkable bogosort(x) setup=(x=rand(n)) size=:n
res = run(b, 10)

would time bogosort with a size n=10. With this, a type BenchmarkSeries which encapsulates a Benchmark or BenchmarkGroup and a vector of sizes could let you investigate the size dependence of a benchmark:

function Base.run(bs::BenchmarkSeries{Benchmark})
    map(bs.sizes) do size
        time(run(bs.benchmarkable, size))
    end
end

This would help you answer questions like "how large should a vector be before algorithm A overtakes algorithm B?".

@vchuravy
Copy link
Member

vchuravy commented Jun 7, 2021

That sounds like a nice addition!

@gdalle
Copy link
Collaborator

gdalle commented Jun 13, 2023

How different is it from creating a BenchmarkGroup with various values of n as keys?

@gustaphe
Copy link
Contributor Author

Functionally similar, interface wise quite different. Say you're comparing four different algorithms, and you want to benchmark them for size 10, 100, 1000, 10_000, etc. Handling this with a benchmarkgroup becomes unwieldy, you can of course loop through and create dynamic keys, but that's a lot of boilerplate. And the display of your result will not automatically respect the inherent structure. A benchmarkseries could have a plot recipe connected to it, a numbered benchmarkgroup not really.

@gdalle
Copy link
Collaborator

gdalle commented Jun 14, 2023

I think this could be achieved by #256 without changing the interface?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants