Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase Shogun coverage in benchmarks #4046

Open
karlnapf opened this issue Dec 23, 2017 · 5 comments
Open

Increase Shogun coverage in benchmarks #4046

karlnapf opened this issue Dec 23, 2017 · 5 comments

Comments

@karlnapf
Copy link
Member

karlnapf commented Dec 23, 2017

This task is to check MLPack's benchmarking framework, and add more Shogun methods to it. We eventually want all of Shogun's algorithms in there to be covered.
This is really quite simple, and requires few scripts to be added, examples

Some information can be found here:
https://github.com/shogun-toolbox/shogun/wiki/GSoC_2017_project_fundamental_usual_suspects

Another nice thing to have would be a sorted list of where shogun does best/worst against other frameworks.

Steps:

  1. Find out which algorithms are covered by the framework (both by shogun and in general)
  2. Go to http://www.mlpack.org/benchmark.html and check the latest reports
  3. Find a reported algorithm that is implemented in Shogun, but not yet covered by the benchmarks (LARS is such an example)
  4. Add it
  5. Goto 1
@pilar12
Copy link

pilar12 commented Dec 25, 2017

I am interested in working on this. Which methods need to be worked on?

@karlnapf
Copy link
Member Author

I have written a little list of steps to identify which method to work on above

@p16i
Copy link
Contributor

p16i commented Feb 7, 2019

I'm taking a look at this issue. I've created a spreadsheet for the current coverage at https://goo.gl/Qv9vxU.

I'll start go down the list, adding methods that are implemented in Shogun but aren't included in the benchmark.

image

Updated: An automatic way to get the results is using the cURL mentioned in #4097.

@karlnapf
Copy link
Member Author

karlnapf commented Feb 8, 2019

This is really useful info. Thanks for that! Curious to see what this leads to performance wise

@zoq
Copy link
Contributor

zoq commented Feb 13, 2019

Agreed, this is really nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants