NASBench

NASBench is a built-in problem based on NAS-Bench-101: Towards Reproducible Neural Architecture Search. The official Python implementation of NAS-Bench-101 can be found here.

How to setup the dataset file for NASBench problem

This problem requires to download a dataset file and convert it to the binary format for kurobako. Please execute the following commands to set up the dataset file before running benchmarks.

// Download dataset file.
$ curl -L $(kurobako dataset nasbench url) -o nasbench_full.tfrecord

// Convert the TFRecord format file to the binary file that can be loaded by kurobako.
$ kurobako dataset nasbench convert nasbench_full.tfrecord nasbench_full.bin

You can specify the dataset file as follows:

// Create a NASBench problem recipe.
$ kurobako problem nasbench nasbench_full.bin

Use NASBench as a two-objective problem

As a default, NASBench problem behaves as a single-objective problem to optimize the validation accuracy. However, you can make it a two-objective problem by specifying --metrics option as follows:

// The objective is to optimize both the validation accuracy (maximize) and the number of model parameters (minimize).
$ kurobako problem nasbench nasbench_full.bin --metrics params accuracy
{"nasbench":{"dataset":"nasbench_full.bin","encoding":"A","metrics":["PARAMS","ACCURACY"]}}