Skip to content
Bayesian Optimization with Neural Architectures for Neural Architecture Search - https://arxiv.org/abs/1910.11858
Python Jupyter Notebook Shell
Branch: master
Clone or download
Latest commit d7b1326 Nov 13, 2019

README.md

BANANAS

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
Colin White, Willie Neiswanger, and Yash Savani.
arXiv:1910.11858.

A new method for neural architecture search

BANANAS is a neural architecture search (NAS) algorithm which uses Bayesian optimization with a meta neural network to predict the validation accuracy of neural architectures. We use a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model. After training on just 200 architectures, we are able to predict the validation accuracy of new architectures to within one percent on average. The full NAS algorithm beats the state of the art on the NASBench and the DARTS search spaces. On the NASBench search space, BANANAS is over 100x more efficient than random search, and 3.8x more efficent than the next-best algorithm we tried. On the DARTS search space, BANANAS finds an architecture with a test error of 2.57%.

bananas_fig

Requirements

  • tensorflow == 1.14.0
  • pytorch == 1.2.0, torchvision == 0.4.0
  • matplotlib, jupyter
  • nasbench (follow the installation instructions here)

You will also need our fork of the darts repo:

  • Download the repo: https://github.com/naszilla/darts
  • If the repo is not in your home directory, i.e., ~/darts, then update line 5 of bananas/darts/arch.py and line 8 of bananas/train_arch_runner.py with the correct path to this repo

Train a meta neural network with a notebook on the NASBench dataset

  • Download the nasbench_only108 tfrecord file (size 499MB) here
  • Place nasbench_only108.tfrecord in the top level folder of this repo
  • Open and run meta_neuralnet.ipynb to reproduce Table 1 and Figure A.1 of our paper

bananas_fig bananas_fig bananas_fig bananas_fig

Evaluate pretrained BANANAS architecture

The best architecture found by BANANAS on the DARTS search space achieved 2.57% test error. To evaluate our pretrained neural architecture, download the weights bananas.pt and put it inside the folder <path-to-darts>/cnn

cd <path-to-darts>/cnn; python test.py --model_path bananas.pt

The error on the test set should be 2.57%. This can be run on a CPU or GPU, but it will be faster on a GPU.

bananas_normal bananas_reduction

The best neural architecture found by BANANAS on CIFAR-10. Convolutional cell (left), and normal cell (right).

Train BANANAS architecture

Train the best architecture found by BANANAS.

cd <path-to-darts>/cnn; python train.py --auxiliary --cutout

This will train the architecture from scratch, which takes about 34 hours on an NVIDIA V100 GPU. The final test error should be 2.59%. Setting the random seed to 4 by adding --seed 4 will result in a test error of 2.57%. We report the random seeds and hardware used in Table 2 of our paper here.

Run BANANAS on the NASBench search space

To run BANANAS on NASBench, download nasbench_only108.tfrecord and place it in the top level folder of this repo.

python run_experiments_sequential.py

This will test the nasbench algorithm against several other NAS algorithms on the NASBench search space. To customize your experiment, open params.py. Here, you can change the hyperparameters and the algorithms to run.

nasbench_plot

Run BANANAS on the DARTS search space

We highly recommend using multiple GPUs to run BANANAS on the DARTS search space. You can run BANANAS in parallel on GCP using the shell script:

run_experiments_parallel.sh

Citation

Please cite our paper if you use code from this repo:

@article{white2019bananas,
  title={BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search},
  author={White, Colin and Neiswanger, Willie and Savani, Yash},
  journal={arXiv preprint arXiv:1910.11858},
  year={2019}
}
You can’t perform that action at this time.