Is NATS Extension of NAS_201 bench

[Mars-204]

I was working with NAS_201 bench earlier and now am shifting to NATS bench. I have the following doubts:

• Will the index of architecture obtained with NAS_201 and NATS bench same for a given arch? Because when using both NAS_201 is giving 12804 index while NATS bench is returning -1. Also the index of NAS_201 provides different config arch when used as index in NATS.
• While obtaining weights I am getting an empty return when using the NAS_201 index.

Following is my implementation. I am using benchmark file with sss.

api = create(d, 'sss', fast_mode=False, verbose=True)
index = api.query_index_by_arch(convert_naslib_to_str(best_arch))
config = api.get_net_config(index, 'cifar10')
best_arch = get_cell_based_tiny_net(config)
logger.info("Queried results ({}): {}".format(metric, best_arch))
params = api.get_net_param(index, 'cifar10', None)
best_arch.load_state_dict(next(iter(params.values())))

[D-X-Y]

Hi @Mars-204 , thanks for your interests.

For the tss space, it is the same index. For the sss (size search space), NAS-Bench-201 does not have the size search space. For the examples, you should compare api = create(d, 'tss', fast_mode=False, verbose=True) with NAS-Bench-201.

[Mars-204]

Hi @D-X-Y , thanks for the reply. It works for tss as you mentioned.

But when I am trying to obtain the pre-trained weights for this index I am getting a an empty dic in return. Currently I am using the benchmark compressed pickle for tss for the same. Should I be using any other files for the pre-trained weights?

d = pickle_load('/work/ws-tmp/g059997-naslib/g059997-naslib-1667607005/NASLib_mod/naslib/NATS-bench/Copy of NATS-tss-v1_0-3ffb9.pickle.pbz2')
api = create(d, 'tss', fast_mode=False, verbose=True)
index = api.query_index_by_arch(convert_naslib_to_str(best_arch))
config = api.get_net_config(index, 'cifar10')
best_arch = get_cell_based_tiny_net(config)
params = api.get_net_param(index, 'cifar10', None)
best_arch.load_state_dict(next(iter(params.values())))

[D-X-Y]

The network parameters are contained in full archive files (please follow the instructions at https://github.com/D-X-Y/NATS-Bench#preparation-and-download). Note that those files are pretty large.

[Mars-204]

Thanks for the response. It is working.