From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Symbolic Regression

Quan Minh Phan, Ngoc Hoang Luong

Setup

$ conda create -n py39 python=3.9
$ conda activate py39
$ pip install -r requirements.txt

All numerical results and figures are stored in folders result and fig, respectively.

Reproducing the results

You can search from scratch to find 31 expression trees as in the paper or utilize the provided trees in the folder exp. To search for expression trees, run the below script (all expression trees would be stored in the folder exp):

$ python gp_search_multiple.py --SR_dataset D

After the process is finished, we will obtain the Table 2 in the paper.

Whenever we have all expression trees (in case you re-conduct the searches), we can reproduce all results in the paper.

• Figure 5:

$ python exp_statistics.py

• Get the best metric (Equation 2) & Figure 6:

$ python evaluate_SR-designed_metric.py --measure kendall

• Ablation Study (Sections 4.5.1. 4.5.2):

$ python gp_search_multiple.py --SR_dataset D+
$ python gp_search_multiple.py --SR_dataset D-
$ python gp_search_single.py --ss nb101
$ python gp_search_single.py --ss nb201
$ python gp_search_single.py --ss nb301
$ python run_ablation_study_451_452.py

• Appendix D (all figures are stored in the folder fig):

$ python run_appendixD.py

• Appendix E (all results are stored in the folder results):

$ python run_appendixE.py