Skip to content

machinelearningnuremberg/DeepRankingEnsembles

Repository files navigation

Deep Ranking Ensembles

Repository for Deep Ranking Ensembles for Hyperparameter Optimization (paper accepted at ICLR 2023).

Setup

Environment Setup

  • Install anaconda/miniconda according to the Installation Instructions.
  • Clone this repository into a folder. git clone --recurse-submodules <url> ./repo
  • Go inside the repo folder. cd repo
  • Create the required conda environment. conda env create --file linux_environment.yml
  • If the environment is already created, update it using the command conda env update --file linux_environment.yml --prune.
  • This creates a conda environment called DRE. Activate the conda environment using command conda activate DRE.
  • After activating DRE environment, display the script usage help message using command python DRE.py -h.
  • Please download the data as described in the next section before running the DRE.py script.

Data Download

  • Download the HPO_B data from HERE. A file named hpob-data.zip will be downloaded.
  • Extract hpob-data.zip to the location ./repo/HPO_B/hpob-data/. Here the ./repo/HPO_B folder is the location where the HPO_B submodule is cloned.
  • After successful extraction, all the required files (in the json format) will be present in the ./repo/HPO_B/hpob-data/ folder.
  • The extracted files should be in the following folder structure:
    • repo
      • HPO_B
        • hpob-data
          • bo-initializations.json
          • meta-test-dataset.json
          • meta-test-tasks-per-space.json (This file is present in the folder by default)
          • meta-train-dataset-augmented.json
          • meta-train-dataset.json
          • meta-validation-dataset.json

Example command

  • To train & evaluate the DeepRankingSurrogate, you can run the following commands chronologically.
    • python DRE.py --train --train_index 0 --meta_features --M 10 --layers 4 --result_folder ./results_M10/. After running this, a file named '4796' which is the search space ID corresponding to index 0 is created in the ./results_M10 folder. Please check the next section.
    • python DRE.py --evaluate --eval_index 0 --meta_features --M 10 --layers 4 --result_folder ./results_M10/. After running this, a file named EVAL_KEY_0 will be created in the ./results_M10 folder.
  • Please make sure that the architecture and the result folder for the training and evaluation is identical.

Index Table

The following table shows the train_index and eval_index corresponding to each search space in the hpob_data. Each train_index refers to a single Search Space ID. On the other hand, each Search Space has a range of evaluation indices. The set of all evaluation indices of a search space is given by the cross product: {Set containing the IDs of all Datasets of the search space} X {Set containing random seeds used to start the BO iteration}. For example eval_index = 0 corresponds to ('4796', '23','test0').

Search Space ID Train Index Eval Index
4796 0 0 - 19
5527 1 20 - 49
5636 2 50 - 79
5859 3 80 - 109
5860 4 110 - 124
5889 5 125 - 134
5891 6 135 - 164
5906 7 165 - 174
5965 8 175 - 209
5970 9 210 - 239
5971 10 240 - 269
6766 11 270 - 299
6767 12 300 - 329
6794 13 330 - 359
7607 14 360 - 394
7609 15 395 - 429

Citation

You can cite our work as follows:

@inproceedings{
khazi2023deep,
title={Deep Ranking Ensembles for Hyperparameter Optimization},
author={Abdus Salam Khazi and Sebastian Pineda Arango and Josif Grabocka},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=_ruvo2KCL2x}
}