Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reporting format of benchmarks #12

Open
amueller opened this issue Jul 5, 2019 · 0 comments · May be fixed by #133
Open

reporting format of benchmarks #12

amueller opened this issue Jul 5, 2019 · 0 comments · May be fixed by #133

Comments

@amueller
Copy link

amueller commented Jul 5, 2019

As discussed here: scikit-learn/scikit-learn#14247 (comment)

I think the current report is very hard to read.
It might be helpful to specify very clearly what the baseline is, that is the meaning of 1 in all the plots - it's your own C++ implementation.

For a comparison with scikit-learn I think doing sklearn speed / your c++ speed would be easier to read as it shows your speedup factor, not our slow-down factor.

Finally, I don't see the number of cores in your benchmark, which is pretty crucial since most of our implementations are single-threaded. Yes, that's a big issue, but saying "we're 100x faster" without saying "on 100 CPUs instead of 1" is quite misleading.
It might be helpful to have a chart of speedup vs number of CPUs.

@Alexsandruss Alexsandruss linked a pull request May 17, 2023 that will close this issue
@Alexsandruss Alexsandruss linked a pull request May 17, 2023 that will close this issue
razdoburdin pushed a commit to razdoburdin/scikit-learn_bench that referenced this issue Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant