The details of results in the paper

[Littleyezi]

Hi, I want to use the NATS-Bench, but I still have some questions about the results of benchmark algorithms given in the paper.
(1) Hp=200 in tss and hp=90 in sss, right?
(2)the results is top1 or top5 or others?
(3)Sorry, but can I trouble you to wirte a usage example to show how can I get the correct metrics when I search a new arch?
Thanks!

[D-X-Y]

Thanks for your interest in NATS-Bench.

(1), Yes, for the full training, hp=200 for tss, hp=90 for ss. For the low-fidelity approximation, both tss and sss use hp=12.

(2) all accuracy is reported for top-1.

(3) Yes, could you mind clarifying what do you want? There are many metrics. It depends on your demand to choose one?

[D-X-Y]

FYI, here is an usage example on how to obtain the test accuracy (that is used to report in the tables in our paper).

xinfo = api.get_more_info(arch, dataset=dataset, hp=90 if is_size_space else 200, is_random=False)
test_accuracy = xinfo["test-accuracy"]

The arch indicate the searched architecture, the dataset indicates one of the three datasets used in our paper, is_size_space is a boolean value indicating whether it is sss or tss.

[Littleyezi]

Thanks for all! I try to do some research about NAS algorithms and want to use the NATS-Bench as dataset,hence I must to ensure correct information about the results so that I can directly compare with the benchmark algorithms in paper. And the validation of a new arch corresponding to in paper is xinfo["valid-accuracy"]?

[D-X-Y]

@Littleyezi For the validation accuracy, it is a little bit difference:

if dataset == "cifar10": 
  xinfo = api.get_more_info(arch, dataset="cifar10-valid", hp=90 if is_size_space else 200, is_random=False)
  valid_acc = xinfo["valid-accuracy"]
else:
  xinfo = api.get_more_info(arch, dataset=dataset, hp=90 if is_size_space else 200, is_random=False)
  valid_acc = xinfo["valid-accuracy"]

[Littleyezi]

@Littleyezi For the validation accuracy, it is a little bit difference:

if dataset == "cifar10": 
  xinfo = api.get_more_info(arch, dataset="cifar10-valid", hp=90 if is_size_space else 200, is_random=False)
  valid_acc = xinfo["valid-accuracy"]
else:
  xinfo = api.get_more_info(arch, dataset=dataset, hp=90 if is_size_space else 200, is_random=False)
  valid_acc = xinfo["valid-accuracy"]

Thanks!

[D-X-Y]

U are welcome