No description, website, or topics provided.
Clone or download
shukon Development (#167)
* FIX logger, add ALWAYS debug to file

* FIX logger, add ALWAYS debug to file

* MAINT add phantomjs to install/ fix args

* ADD phantomjs-requirement to installation-docs

* MAINT rework configuration footprints (most of infrastructure, add development over time)

* WIP better slider

* ADD static cfp-plots, ADD cli-option to toggle performance-tradeoff in timeslider

* FIX error, set default for timeslider

* RM test-tmp-files

* MAINT improve logs and docs, refactor cfp

* MAINT docs, fix typos, add phantomjs to circle-build(?)

* MAINT docs, fix typos, add phantomjs to circle-build(?)

* FIX bugs regarding cfp-refactoring

* FIX tests (use extra example with fewer features)

* ADD missing test-files

* ADD missing test-files

* FIX requirements and circle script

* ADD implement max_plot in cfp (finally)

* ADD fallback-msg if plots are not exported in bokeh

* MAINT only use one rh in cfp

* RM online-timeslider for cfp, leaving only on or off

* ADD max_plot, fix deps, rm singletons from cfp

* FIX tests

* MAINT improve comments and minor changes

* MAINT last refactoring, small bugfix in contourplot

* Issue 56 expand stat tests on metrics (#151)

* ADD permutation test to performance table for all metrics

* MAINT rm p-values from meta-data and fix tooltips

* FIX get_costs to be compatible with smac-master

* ADD number changed parameters to overview

* FIX typing in html-builder

* FIX order of algorithm footprints

* MAINT enforcing order

* ADD save-button to bokeh-plots

* MAINT refactor cost-over-time and make plotter and cost-over-time flake8-compliant

* FIX missing args through refactor of cot

* FIX bugs: reenable confviz-toggle, only plot timeslider if num-quants > 1

* Issue 152 smac2 validation (#153)

* ADD validation-format

* FIX validation load smac2

* ADD tests for smac2 reader

* MAINT req back to smac==080

* FIX different validation format

* Issue 72 algorithm footprints mouseover (#154)

* MAINT improve tooltips

* ADD interactive algorithm footprint (incl. tooltips)

* WIP

* RM algo-2d plots

* Prepare release 1.1.0 (#156)

* FIX missing feat-dict-exceptions

* FIX bugs --source-- and too few params in parallels and add tooltip to forsel

* MAINT clarify timeout-scores

* ADD extensive cornercase test-script (testing with, without instances, features, etc)

* MAINT refactor, more verbose runhistory-handling, PEP8, move plotters respons. into analyzer, overall improvement

* UPDATE docs, add full api-doc

* UPDATE example

* FIX performance-table (add oracle)

* FIX color scale in correlation plot

* ADD version number to html report

* MAINT improve tables

* MAINT algo footprint tools

* MAINT pep8, extensive tests

* ADD missininstanceserror

* FIX smac-version to dev-commit

* VERSION 1.1.0

* FIX smac to >=0.8.0 and add notice to FAQ, also update readme

* MAINT release 1.1.0

* Update __version__.py

* Add bohb support (#158)

Add BOHB-support. --file_format now accepts BOHB which will run CAVE for every budget explored in the optimization-process of BOHB.

Bugfixes:

    catch fANOVA error on too few datapoints
    add sanity checks for exceptional cases (e.g. with very few configs)
    bind memory error in parallel coordinates by limiting configs available for epm-training
    small bugfixes

* FIX catch runtime error if fanova has too few datapoints

* ADD small sanity checks for exceptional cases with very few configs

* ADD all lpi-plots - maybe not so good

* ADD support for BOHB

* FIX performance table (only display relevant)

* ADD informative error when encountering fanova error

* FIX error on missing cutoff

* UPDATE readme, especially cmdline-args

* FIX memory error (hopefully) in parallel coordinates by limiting number of train-confs for epm

* MAINT throw error on unknown args to avoid typos

* FIX error in pc on negative costs, now only using log_costs on runtime-data

* FIX tests

* FIX parallel coordinate limits

* MAINT small bugfixes

* FIX validation data smac3

* FIX validation data smac3

* FIX tests

* Add fanova std (#159)

* FIX catch runtime error if fanova has too few datapoints

* ADD small sanity checks for exceptional cases with very few configs

* ADD all lpi-plots - maybe not so good

* ADD support for BOHB

* FIX performance table (only display relevant)

* ADD informative error when encountering fanova error

* FIX error on missing cutoff

* UPDATE readme, especially cmdline-args

* FIX memory error (hopefully) in parallel coordinates by limiting number of train-confs for epm

* MAINT throw error on unknown args to avoid typos

* FIX error in pc on negative costs, now only using log_costs on runtime-data

* FIX tests

* FIX parallel coordinate limits

* MAINT small bugfixes

* FIX validation data smac3

* FIX validation data smac3

* FIX tests

* ADD uncertainty to fanova

* MAINT manage fanova imps

* MAINT catch exception on bokeh export

* FIX tests + backwardscompatibility

* Improve bohb report (#160)

* ADD BOHB-support in one html, changes in constructors

* FIX hpbandster converter and update readme

* Update changelog

* ADD bohb-example

* FIX double validation format error

* INCREASE version dev

* Add Jupyter & BOHB support (#163)

* INCREASE version dev

* ADD jupyter support, make analyzer an abstract class for analyzing methods

* ADD bohb learning curves visualization

* PREPARE RELEASE for jupyter-support

* RM old import

* DEV VERSION

* MAINT documentation (merge tooltips and docstrings, add documentation for jupyter and budgets) (#165)

* Notebookfixes (#166)

* MAINT make performance table flexible, show only what is needed

* FIX automl-signature + logo

* ADD tabs for budgets

* FIX cmd-line toggle of cfp-timeslider

* ADD explanation to missing plot (alt text), add budgets to docs

* RM irrelevant info from overview-table

* ADD axis labels to bohb plot

* RM Performance Table tag

* ADD color-selection in bohb-plot

* FIX adjust bohb-plot-widget-sizes

* FIX verbose OFF filters everything
Latest commit 64b541e Oct 9, 2018

README.md

Status for master branch / development branch:

Build Status / Build Status

CAVE

CAVE is a versatile analysis tool for automatic algorithm configurators. It generates comprehensive reports (e.g. http://ml.informatik.uni-freiburg.de/~biedenka/cave.html) that give you insights into the configured algorithm, the used instance set and also the configuration tool itself. The current version works out-of-the-box with SMAC3, but can be easily adapted to other configurators, as long as they use the same output-structure. You can also find a talk on CAVE online.

LICENSE

Please refer to LICENSE

If you use out tool, please cite us:

    @InProceedings{biedenkapp-lion18a,
        author = {A. Biedenkapp and J. Marben and M. Lindauer and F. Hutter},
        title = {{CAVE}: Configuration Assessment, Visualization and Evaluation},
        booktitle = {Proceedings of the International Conference on Learning and Intelligent Optimization (LION'18)},
        year = {2018},
        month = jun}

OVERVIEW

CAVE is an analysis tool. It is written in Python 3.5 and uses SMAC3, pimp and ConfigSpace. CAVE generates performance-values (e.g. PAR10), scatter- and cdf-plots to compare the default and the optimized incumbent and provides further inside into the optimization process by quantifying the parameter- and feature-importance.

REQUIREMENTS

  • Python 3.5
  • SMAC3 and all its dependencies
  • ParameterImportance and all its dependencies
  • everything specified in requirements.txt Some of the plots in the report are generated using bokeh. To automagically export them as .pngs, you need to also install phantomjs-prebuilt. CAVE will run without it, but you will need to manually export the plots if you wish to use them.
  • phatomjs-prebuilt

INSTALLATION

You can install CAVE from pip:

pip install cave

or clone the repository and install requirements into your virtual environment.

git clone https://github.com/automl/CAVE.git && cd CAVE
pip install -r requirements.txt

To have some .pngs automagically available, you also need phantomjs.

npm install phantomjs-prebuilt

USAGE

We are currently working on the documentation of CAVE. Here a little Quickstart-Guide.

You can analyze multiple folders (that are generated with the same scenario) for the analysis, simply provide the paths to all the individual results in --folders.

Commandline arguments:

  • --folders: path(s) to folder(s) containing the SMAC3-output (works with output/run_*)

Optional:

  • --output: where to save the CAVE-output
  • --file_format: of results to be analyzed, choose from SMAC3, SMAC2, CSV or BOHB
  • --validation_format: of (optional) validation data (to enhance epm-quality where appropriate), choose from SMAC3, SMAC2, CSV or NONE
  • --ta_exec_dir: target algorithm execution directory, this should be a path to the directory from which SMAC was run initially. used to find instance-files and if necessary execute the algo-parameter of the SMAC-scenario (DEFAULT: current working directory)
  • --parameter_importance: calculating parameter importance is expensive, so you can specify which plots you desire: ablation, forward_selection, fanova and/or lpi. either provide a combination of those or use all or none
  • --feature_analysis: analysis features is expensive, so you can specify which algorithm to run: box_violin, clustering, importance and/or feature_cdf. either provide a combination of those or use all or none
  • --no_tabular_analysis: toggles the tabular analysis
  • --no_ecdf, --no_scatter_plots: toggle ecdf- and scatter-plots
  • --no_cost_over_time: toggles the cost-over-time plot
  • --no_parallel_coordinates: toggles the parallel-coordinates plot
  • --no_configurator_footprint: toggles the configurator-footprints
  • --no_algorithm_footprints: toggles the algorithm-footprints
  • --cfp_time_slider: how to display the over-time development of the configurator footprint, choose from off (which yields only the final interactive plot), static (which yields a number of .pngs to click through), online (which generates a time-slider-widget - might be slow interaction on big data) and prerender (which also generates time-slider, but large file with low interaction time)
  • --cfp_number_quantiles: if time-slider for configurator footprints is not off, determines the number of different quantiles to look at

For further information on to use CAVE, see: `python scripts/cave.py -h

EXAMPLE

You can run the spear-qcp example like this:

cave --folders examples/smac3/example_output/* --ta_exec examples/smac3/ --output CAVE_results/

This will analyze the results located in examples/smac3 in the dirs example_output/run_1 and example_output/run_2. The report is located in CAVE_results/report.html. --ta_exec corresponds to the folder from which the optimizer was originally executed (used to find the necessary files for loading the scenario). For other formats, e.g.:

cave --folders examples/smac2/ --ta_exec_dir examples/smac2/smac-output/aclib/state-run1/ --file_format smac2 --no_algorithm_footprint
cave --folders examples/csv_allinone/ --ta_exec_dir examples/csv_allinone/ --file_format csv

USAGE WITH BOHB

You can also use cave with Configurators that use budgets to estimate a quality of a certain algorithm (e.g. epochs in neural networks), a good example for this behaviour is BOHB. To interpret BOHB's results, you need to install an additional dependency:

pip install hpbandster

and then you can use CAVE as usual, specifying the file_format as BOHB:

cave --folders examples/bohb --file_format BOHB --output CAVE_BOHB_results

There is an example jupyter-notebook on how to use CAVE with BOHB.