There are some problems with the accuracy of the model

[tangtaogo]

Which Algorithm
Size Search Space
Cifar 10

Describe the Question
I get all 32768 architecture candidates‘ acc using the code followed, but all of them in under 90%.
The results in your paper of all method is above 90%. How can I get the result of table 4 in your paper?

from nats_bench import create
model_cifar10_rank = {}
api = create(None, 'sss')
for index in range(32768):
info = api.get_more_info(index, 'cifar10')
config = api.get_net_config(index, 'cifar10')
model_cifar10_rank[config['channels']] = info['test-accuracy']

models_cifar10_acc_rank 2.txt

[D-X-Y]

@Trent-tangtao Hi, thanks for using NATS-Bench.
This is because the default hyperparameter kwargs for get_more_info is hp=12 (performance of 12 epoch training). You need to use hp=90, if you want to get the performance of full training.
I just added an example for you, please see: https://github.com/D-X-Y/NATS-Bench/blob/main/notebooks/issue-11.ipynb

[tangtaogo]

Got it, thank you! And thanks for your excellent work！

[Littleyezi]

Hi, so if I want to get the results in tss space like paper, I need to use hp=200, right? But it's said hp=12 in paper, I am not sure if I misunderstood. And when I search a new arch, I just call the simulate_train_eval() function to get the vali_accuracy?

[D-X-Y]

@Littleyezi hp=12 is used during searching to obtain the validation accuracy. hp=200 is used to report the final performance of the searched architecture.
After the searching procedure complete, it is not suggested to call simulate_train_eval.