[AAAI2023] NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension (PDF)(slides)(知乎)

1. Environment

1.1 Install Hyperbox

Our project is based on AutoML Framework hyperbox, so you need to first install hyperbox.

There are two ways to install hyperbox:

• install via pip

pip install hyperbox

• install via git (recommended)

git clone https://github.com/marsggbo/hyperbox
cd hyperbox
python setup.py develop
python install -r requirements.txt
pip install pytorch-lightning==1.5.8 # higher version are not tested

1.2 Prepare enviroment for this project

git clone https://github.com/marsggbo/NAS-LID
cd hyperbox_app
python setup.py develop
pip install -r requirements.txt

2. Code Structure

• hyperbox_app/hyperbox_app/distributed/Imagenet_train: finetune the final searched models on the ImageNet. Please see hyperbox_app/hyperbox_app/distributed/Imagenet_train/scripts for details of finetuning.
• The other folders and files in hyperbox_app/hyperbox_app/distributed are used for Supernet Partition and Evolution-based Search.
• All configurations are managed by hydra library, you can read or modify them in hyperbox_app/hyperbox_app/distributed/configs. An experiment needs to specify multiple modules, such as datamodule, engine, model, trainer, and logger. Specifically, the yaml files in hyperbox_app/hyperbox_app/distributed/configs/experiment integrate all above modules. For example, hyperbox_app/hyperbox_app/distributed/configs/experiment/fewshot_search_nb201.yaml is used for experiment of splitting NB201 search space. You can create you own yaml file to run a different experiment.

3. LID/Gradient-based Supernet Partition

Splitting NAS-Bench-201

3.1 Single-GPU

bash ./scripts/fewshot/fewshot_search_nb201.sh 0 nb201_c10_ID_mincut_sp_16splits \
"logger.wandb.offline=True trainer.strategy=null engine.split_criterion=ID engine.split_method=mincut"

• nb201_c10_ID_mincut_sp_16splits is the experiment name, which is used for creating experiment log folder.
• replacing engine.split_criterion=ID by engine.split_criterion=grad will split the Supernet via gradient.

3.2 Multi-GPU (e.g., 4 GPUs)

bash ./scripts/fewshot/fewshot_search_nb201.sh 4 nb201_c10_ID_mincut_sp_16splits \
"logger.wandb.offline=True trainer.strategy=null engine.split_criterion=ID engine.split_method=mincut +trainer.strategy=ddp"

3.3 For debug:

bash ./scripts/fewshot/fewshot_search_nb201.sh 0 debug_nb201_c10_ID_mincut_sp_4splits \
"ipdb_debug=False logger.wandb.offline=True trainer.strategy=null +trainer.limit_val_batches=10 engine.split_criterion=ID engine.split_method=mincut engine.is_single_path=True +trainer.fast_dev_run=True engine.warmup_epochs=[1,2] engine.repeat_num=1 engine.finetune_epoch=1"

4. Evolutionary Algorithm-based Search

Simialrly, the following line can generate a series of running commands for evolution-based search.

The following two paths are important:

• 1. the configuration template of ea-based search, e.g., /path/to/scripts/fewshot/ea_search_cfgs (absolute path is preferred)
• 2. the log path of the previous experiment in Step 3, e.g., /path/to/logs/runs/nb201_c10_ID_mincut_sp_16splits/2022-12-10_07-50-42

path1=/path/to/scripts/fewshot/ea_search_cfgs
path2=/path/to/logs/runs/nb201_c10_ID_mincut_sp_16splits/2022-12-10_07-50-42
bash scripts/fewshot/ea_search.sh [0] eaSearch_exp_name 1.0 ${path1} " ++engine.supernet_mask_path_pattern=${path2}/checkpoints/*mask.json ipdb_debug=False trainer.fast_dev_run=False model.mutator_cfg.evolution_epochs=100  model.mutator_cfg.population_num=50"

• Command for debug

path1=/path/to/scripts/fewshot/ea_search_cfgs
path2=/path/to/logs/runs/nb201_c10_ID_mincut_sp_16splits/2022-12-10_07-50-42
bash scripts/fewshot/ea_search.sh [0] eaSearch_exp_name 1.0 ${path1} " ++engine.supernet_mask_path_pattern=${path2}/checkpoints/*mask.json ipdb_debug=True trainer.fast_dev_run=True model.mutator_cfg.evolution_epochs=2  model.mutator_cfg.population_num=10"

5. Evaluation on Searched Models

The way of reproducing the results on OFA and ProxylessNAS search space is detailed in the following folder

cd hyperbox_app/hyperbox_app/distributed/Imagenet_train/scripts

You may need to modify the path arguments to your local path in each bash scripts before evaluations.