Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

README.md

OpenML-ASLib

Library that converts a set of experiments from OpenML into an ASLib scenario. The experiments are defined by either

  • A study id
  • A set of tasks and setup ids
  • A benchmark suite and a set of scikit-learn classifiers

The remainder of this tutorial requires knowledge about basic OpenML components, such as datasets, tasks, flows and setups. For an introduction to these concepts in the context of OpenML, please read this blog post.

How to generate an ASLib scenario

There are several ways of using OpenML to generate new ASLib scenario's.

Use an existing study

An OpenML study is a collection of datasets, tasks, flows, setups and runs. Studies bundle these components in a single URL, convenient for attaching to a paper. For example, this study contains all algorithm runs that were used for the OpenML-WEKA-2017 scenario.

The OpenML-ASLib extension will take all registered tasks and setups from this study, and will try to find all corresponding runs on this cross-product on OpenML. Note that it is suggested to have the complete grid of algorithm performances (i.e., all setups should have been ran on all tasks).

Run example: python examples/generate_from_study.py --measure predictive_accuracy --study_id 34 This generates a copy of the OpenML-WEKA-2017 scenario.

Alternatively, if the task ids and setup ids are already known (which requires some moderate OpenML skills) it is also possible to generate a scenario using these: python examples/generate.py --measure predictive_accuracy --task_ids 1701 1702 1705 --setup_ids 2361 2362

Generate runs on the fly

The example script generate_from_sklearn.py shows how to run several scikit-learn on an benchmark suite (= all tasks in an OpenML study) and create a ASLib scenario from this. In the current example, a Decision Stump and Naive Bayes are ran on all tasks in the OpenML-CC18, a curated benchmark suite. However, the script can be easily adapted to run on any benchmark suite or combination of scikit-learn classifiers. When the result of a given setup/task is already present on OpenML, the script will skip this combination and reuse the earlier result.

About

Turns an OpenML study into an ASLib scenario

Resources

License

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.