Missing data dumping code for NAS-Bench-201 in gates.

[ShunLu91]

Dear authors,
Thanks for this nice work.
I try to run the experiments on NAS benchmarks of GATES. However, I found that there was no data dumping code for NAS-Bench-201 though I have successfully repeated the experiments on NAS-Bench-201 for GATES. Specifically, how to generate the PKL file 'nasbench201.pkl' and 'nasbench201_valid.pkl' when running the scripts/nasbench/train_nasbench201_pkl.py?
Look forward to your reply.
Shun Lu

[walkerning]

Hi, thanks for your attention on our work. I try to find this python script on my server, it seems like I use this script:

# -*- coding: utf-8 -*-
# pylint: disable-all
import pickle
import numpy as np
from aw_nas.common import get_search_space
from nas_201_api import NASBench201API as API

search_space = get_search_space("nasbench-201")
num_arch = len(search_space.api)
print("Number of arch data: {}".format(num_arch))
api = search_space.api

# valid_ratio = 0.1
valid_ratio = 0.5
num_valid = int(num_arch * valid_ratio)
train_data = []
for i in range(num_arch - num_valid):
    # first use cifar10!
    # all 200 epoch, epoch maybe use epoch 150 is better.. let's try
    arch_res = api.query_by_index(i)
    arch = API.str2matrix(arch_res.arch_str)
    accs = [[res.eval_acc1es["ori-test@99"], res.eval_acc1es["ori-test@149"], res.eval_acc1es["ori-test@199"]] for res in arch_res.query("cifar10").values()]
    test_accs = np.mean(accs, axis=0) / 100.
    train_data.append((arch, test_accs[2], test_accs[1]))

valid_data = []
for i in range(num_arch - num_valid, num_arch):
    # first use cifar10!
    # all 200 epoch, epoch maybe use epoch 150 is better.. let's try
    arch_res = api.query_by_index(i)
    arch = API.str2matrix(arch_res.arch_str)
    accs = [[res.eval_acc1es["ori-test@99"], res.eval_acc1es["ori-test@149"], res.eval_acc1es["ori-test@199"]] for res in arch_res.query("cifar10").values()]
    test_accs = np.mean(accs, axis=0) / 100.
    valid_data.append((arch, test_accs[2], test_accs[1]))


with open("nasbench201_05.pkl", "wb") as wf:
    pickle.dump(train_data, wf)
with open("nasbench201_05_valid.pkl", "wb") as wf:
    pickle.dump(valid_data, wf)

You'll need NAS-Bench-201 installed, and GATES uses the very first version of the NAS-Bench-201 repository (only this very first version is available at that time), the name of the data file is NAS-Bench-201-v1_0-e61699.pth.

[ShunLu91]

Thanks for such in-time reply.
Though some lines of the code need to be modified to adapt with the newest NAS-Bench-201 API, it solved my problem just right!
Great thanks and best wishes.