Customize NAS Network

Network

BaseNASNetwork

hyperbox.networks.base_nas_network.BaseNASNetwork

properties

• mask
• arch

functions

def arch_size(self, datasize: Optional[Union[Tuple, List]] = None, convert: bool = True, verbose: bool = False) -> Tuple(float, float)

返回当前被激活的子模型的 FLOPs (M) 和 Size (MB)

• datasize: input data size, e.g., datasize=(2,3,64,64)

def build_subnet(self, mask: Union[str, dict], preserve_weight: bool=True): -> nn.Module

• Args:

  • mask: 如果是str类型，则应该是保存的json文件路径，否则应该传入一个dict
  • preserve_weight: 若为True则会从Supernet中继承对应模块的权重

• Return a subnet

def load_from_supernet(self, state_dict, **kwargs)

该方法的适用场景是假设你初始化了一个子网，然后想从supernet中拷贝对应模块的权重，示例如下

from hyperbox.networks.nasbench201 import NASBench201Network
from hyperbox.mutator import RandomMutator

supernet = NASBench201Network()
rm = RandomMutator(supernet)
rm.reset()
mask = rm.export()

# way1: build a subnet
subnet1 = NASBench201Network(maks=mask)
subnet1.load_from_supernet(supernet.state_dict())

# way2: build a subnet from supernet
subnet2 = supernet.build_subnet(mask)

def get_module_by_name(self, name: str) -> nn.Module

根据名字返回对应的Module

def assign_name2modules(self): -> return None

给模型所有modules赋予一个新的属性autoname，其值是该模块在由pytorch自动生成的名字。

Example:

class MyNet(BaseNASNetwork):
    def __init__(self):
        super(MyNet, self).__init__()
        self.op = nn.Sequential(OrderedDict([
            ('op1', nn.Conv2d(3, 64, 3, 1, 1)),
            ('op2', nn.Conv2d(3, 64, 5, 1, 2)),
            ])
net = MyNet()
net.assign_name2modules()
op1 = net.op[0]
print(op1.autoname) # ==> op1
net.get_module_by_name('op1') is op1 # ==> True

Design a new NAS-based Network

1. create a new hyperbox_app/my_app/my_net.py

import torch.nn as nn
from hyperbox.networks.base_nas_network import BaseNASNetwork
from hyperbox.mutables.spaces import OperationSpace

class MyNet(BaseNASNetwork):
    def __init__(self, ks_list=[3,5,7], mask=None):
        super(MyNet, self).__init__(mask)
        ops = []
        for ks in  ks_list:
            ops.append(nn.Conv2d(3, 16, ks, 1, padding=ks//2, bias=False))
        self.ops = OperationSpace(candidates=ops, key='conv_op', mask=self.mask)

    def forward(self, x):
        return self.ops(x)

2. create a new yaml config hyperbox_app/my_app/configs/model/network_cfg/my_net.yaml

_target_: hyperbox_app.my_app.my_net.MyNet
ks_list: [3,5]
mask: null

3. setup experiment yaml file my_exp.yaml

defaults:
  - override /model: random_model.yaml
  - override model/network_cfg: my_net.yaml
  ...

seed: 12345

trainer:
  min_epochs: 1
  max_epochs: 10

4. run

suppose the path to your app is /abs/to/hyperbox_app/my_app

python -m hyperbox.run \
hydra.searchpath=[file:///abs/to/hyperbox_app/my_app/configs] \
experiment=my_exp