Question about calculating MAdds of dynamic network in the paper

[sseung0703]

Thank you for your great work, and I have a question about how to calculate MAdds in your paper.
The dynamic network has different widths and MAdds for each instance, but you denoted MAdds for your networks.
Are they the average MAdds for the whole dataset?

[changlin31]

Yes, the MAdds are averaged over the validation set, e.g.:

DS-Net/dyn_slim/apis/train_slim_gate.py

Line 287 in 2ae9455

flops_m.update(reduced_flops.item(), input.size(0))

[sseung0703]

Thank you for your rapid answer. 😊
I think it is hard to know how much MAdds is required before the validation, but you achieved similar MAdds to comparative algorithms.
Is there any rule to achieve specific MAdds?

[changlin31]

Thanks for the valuable question.
The gate is actually very sensitive to hyperparameters. To avoid troublesome tuning of the loss balancing factors and other hyperparameters, we use different routing space for different network, e.g. we only use the slimmest few sub-networks to form the DS-Net-S.

This can be find in Appendix.A of the paper

In practice, We first test the MAdds for each sub-networks, then manually choose those sub-networks with MAdds around the target MAdds as the dynamic routing space.