get_cost_info(hp="200") returns weird value for some models in topology search space

[mzsrkeen10]

Describe the bug
For some architectures, get_cost_info() returns different flops/params for hp="12" and hp="200" in topology search space.
For example, the number of parameters (for CIFAR-10) of architecture |skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2| is 0.802426 for hp="12", but for hp="200", the value is 0.6403993333333333.

I've found many other architectures in which the same behavior occurs, but some specific examples are as follows:

• |nor_conv_1x1~0|+|nor_conv_3x3~0|skip_connect~1|+|none~0|skip_connect~1|none~2|
• |nor_conv_3x3~0|+|skip_connect~0|skip_connect~1|+|none~0|skip_connect~1|none~2|

To Reproduce

from nats_bench import create

arch = "|skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2|"
api_path = "NATS-tss-v1_0-3ffb9-simple"
search_space = "tss"
dataset = "cifar10"

api = create(api_path, search_space, fast_mode=True, verbose=False)
idx = api.query_index_by_arch(arch)
info_12 = api.get_cost_info(idx, dataset, hp="12")
info_200 = api.get_cost_info(idx, dataset, hp="200")
print(info_12)
print(info_200)

{'flops': 113.95137, 'params': 0.802426, 'latency': 0.016719988563604522, 'T-train@epoch': 21.51544686158498, 'T-train@total': 258.1853623390198, 'T-ori-test@epoch': 1.496117415882292, 'T-ori-test@total': 17.953408990587505}
{'flops': 90.35841, 'params': 0.6403993333333333, 'latency': 0.016719988563604522, 'T-train@epoch': 21.515446861584987, 'T-train@total': 4303.0893723169975, 'T-ori-test@epoch': 1.4961174158822923, 'T-ori-test@total': 299.22348317645844}

• OS: Ubuntu 20.04.3 LTS (Focal Fossa)
• Python version: 3.8.12
• PyTorch version: 1.8.1+cu102

Expected behavior
I think flops/params for hp="12" and hp="200" should be the same.
(From reading the paper and your implementation in AutoDL-Projects, I could not find any factors that would make a difference.)

[D-X-Y]

Thanks for pointing out this issue! I will have a look today.

[D-X-Y]

Hi @mzsrkeen10, I have updated the api to fix this issue.
You can now re-install this library and have a try. Please first uninstall all previous versions, by using pip uninstall nats_bench, and then reinstall it pip install nats_bench==1.6.

I also written a notebook to test the consistent of flops/params for all architectures in both tss and sss at https://github.com/D-X-Y/NATS-Bench/blob/main/notebooks/issue-36.ipynb.

Please let me know if you have further quetions.

[mzsrkeen10]

Hi, now it's working as expected. Thanks for your quick response!

[D-X-Y]

You are welcome! Glad it helps~