https://dl.acm.org/doi/10.5555/3618408.3620126
https://arxiv.org/abs/2306.06852

All the evaluation datasets are canonical benchmark datasets in vision recognition and will be automatically downloaded by torchvision so that the benchmark can be used almost out of box.


For search all baselines: 
cd search;./train_search_all.sh
The search phase will iterate through all the algorithms in the benchmark searching on 3 datasets (CIFAR-10, CIFAR-100, SVHN).
Each dataset is trialed 5 times independently with diffrent seeds for each baseline
The network architecture files, ending with "pl", obtained during the search phase will be placed in the directory of the baseline.
For example
"20211010-140512-concate-sum-darts-bilevel-cifar10.pl" is the architecture file obtained by DARTS searching on CIFAR-10


For evaluation on <dataset>:
cd eval/<dataset>/;./train.sh <arch file dir> <log file header>
example: cd eval/cifar10; ./train.sh arch/cifar10/pcdarts pcdarts_cifar10
Evaluate all arch files within 'arch/cifar10/pcdarts' directory in the order of Base, 1M, 3ops, 4out.
'pcdarts_cifar10' is used to name the evaluation log. Logs of Base, 1M, 3ops, and 4out will be recorded separately in pcdarts_cifar10_ori.log, pcdarts_cifar10_1M.log, pcdarts_cifar10_3op.log, pcdarts_cifar10_4out.log.


To further enhance reproducibility, we also provide the architecture files used in our evaluation of the benchmark in the paper in results/<dataset>/<method>(ended with "pl"). These files are obtained from multiple trials of the search on the <dataset> on LHD for the corresponding <method>.