In this repository we release all code to replicate all results, tables and figures presented in the paper: HPO X ELA: Investigating Hyperparameter Optimization Landscapes by Means of Exploratory Landscape Analysis
The repository is structured as follows:
data/
contains raw benchmark data and preprocessed data used for analyses. Note that you can also download the data via syncshare: https://syncandshare.lrz.de/getlink/fiNzyFM7s7jGkCEAzTyjS8/dataplots/
contains plots as presented in the papertasks/
contains data of all HPO tasksela_splits.csv
contains the exact CV splits used for the HPO problemsrun_hpo.R
andrun_bbob.R
contain code to run optimizers on HPO and BBOB problems;run_gensa_ablation
contains code to run the GENSA ablation study (see appendix)- Job scheduling on HPCs was performed using batchtools
- If you are interested in the full batchtools registries with all available information, please open an issue
compute_features_hpo.R
andcompute_features_bbob.R
contain code for computing ELA features on HPO and BBOB problemspreprocess_hpo.R
contains code to preprocess HPO data and visualize surface landscapesoptimizer_performance.R
contains code for the analysis of optimizer performanceoptimizer_performance_gensa.R
contains code for the analysis of the GENSA ablation study (see appendix)ert.R
contains code for the ERT analyses of optimizersela_analysis.R
contains code for the analysis of ELA featuresela_cluster.R
contains code for the cluster analysis of ELA featurespredict_kmeans.R
contains helper code for predicting in k-means clusteringtasks_cv_splits.R
contains code to generate the CV splits used for the HPO problemsrenv.lock
lists the exact R packages that were used on the cluster and can be used for setting up an renv environmentappendix.pdf
is our online appendix
You can find our appendix here.