survey-zero-shot-nas

Fig.1 - Comparision of various NAS approaches

Paper list of zero-shot NAS

• Zero-Cost Proxies for Lightweight NAS
• Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search
• LiteTransformerSearch: Training-free On-device Search for Efficient Autoregressive Language Models
• Training-free Transformer Architecture Search
• Training-Free Hardware-Aware Neural Architecture Search with Reinforcement Learning
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• A Feature Fusion Based Indicator for Training-Free Neural Architecture Search
• Neural Architecture Search without Training
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Revisiting Efficient Object Detection Backbones from Zero-Shot Neural Architecture Search
• A Training-Free Genetic Neural Architecture Search
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• How does topology influence gradient propagation and model performance of deep networks with DenseNet-type skip connections?
• FLASH: Fast Neural Architecture Search with Hardware Optimization
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis
• Reducing Neural Architecture Search Spaces with Training-Free Statistics and Computational Graph Clustering
• EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
• Zero-Cost Proxies Meet Differentiable Architecture Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Training-Free Multi-objective Evolutionary Neural Architecture Search via Neural Tangent Kernel and Number of Linear Regions
• Extensible Proxy for Efficient NAS
• ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients

How to use this repo

This repo is designed to evaluate different zero-shot proxied for various benchmarks.

Install

git clone https://github.com/google-research/nasbench
cd nasbench

In the nasbench folder, you need to modify import tensorflow as tf into import tensorflow.compat.v1 as tf for the following files：

   nasbench/api.py
   nasbench/lib/evaluate.py
   nasbench/lib/training_time.py

Then install nasbench pip install -e .

Data and Benchmark download

Go to ~/dataset/img16/ImageNet16/

• Download ImageNet16-120: gdown https://drive.google.com/uc?id=1vZe9VD0Sv5kTw-lR5lT-cmjBSh4AuLAH

Go to ~/dataset/nasbench/:

• Download NASBench-101: wget https://storage.googleapis.com/nasbench/nasbench_full.tfrecord
• Download NASBench-201: gdown https://drive.google.com/uc?id=16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_

Go to ~/dataset/nasbench/NATS

• Download NATS-Bench-SSS: gdown https://drive.google.com/uc?id=1scOMTUwcQhAMa_IMedp9lTzwmgqHLGgA

• Optional (since NATS-Bench-TSS is same as NASBench-201):

  • Download NATS-Bench-TSS: gdown https://drive.google.com/uc?id=17_saCsj_krKjlCBLOJEpNtzPXArMCqxU

Usage

python main.py --searchspace=$SEARCH_SPACE --dataset=$DATASET --data_path=$PATH_OF_DATASET --metric=$PROXY_NAME