[Feature] Add Dsnas Algorithm #226

gaoyang07 · 2022-08-10T09:54:53Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Add Dsnas algorithm.

Modification

Add Dsnas algorithm.
Add mutable.OneHotMutableOP for Dsnas.
Use estimator (pr open-mmlab/mmrazor#230) to get subnet FLOPs.
Add early fix arch mechanism.
Align lr & param scheduler settings for Dsnas.
Update UT & Doc-string.

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

1. finalized compute_loss & handle_grads in algorithm; 2. add MMRAZOR_EVALUATOR; 3. fix bugs.

codecov · 2022-08-11T11:12:42Z

Codecov Report

Base: 0.41% // Head: 0.42% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (d676d93) compared to base (e3390ce).
Patch coverage: 0.00% of modified lines in pull request are covered.

❗ Current head d676d93 differs from pull request most recent head a3f7017. Consider uploading reports for the commit a3f7017 to get more accurate results

Additional details and impacted files

@@           Coverage Diff            @@
##           dev-1.x    #226    +/-   ##
========================================
  Coverage     0.41%   0.42%            
========================================
  Files          174     167     -7     
  Lines         6952    6841   -111     
  Branches      1119    1128     +9     
========================================
  Hits            29      29            
+ Misses        6918    6807   -111     
  Partials         5       5

Flag	Coverage Δ
unittests	`0.42% <0.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmrazor/engine/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/engine/optimizers/__init__.py	`0.00% <0.00%> (ø)`
...razor/engine/optimizers/dsnas_optimizer_wrapper.py	`0.00% <0.00%> (ø)`
mmrazor/models/algorithms/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/models/algorithms/nas/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/models/algorithms/nas/dsnas.py	`0.00% <0.00%> (ø)`
...rchitectures/backbones/searchable_shufflenet_v2.py	`0.00% <0.00%> (ø)`
mmrazor/models/mutables/__init__.py	`0.00% <ø> (ø)`
mmrazor/models/mutables/mutable_module/__init__.py	`0.00% <ø> (ø)`
...els/mutables/mutable_module/diff_mutable_module.py	`0.00% <0.00%> (ø)`
... and 28 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

1. update param_scheduler & lr_scheduler for dsnas; 2. fix a bug of switching to finetune stage.

sunnyxiaohu · 2022-08-10T10:43:31Z

mmrazor/models/mutables/mutable_module/diff_mutable_module.py

@@ -58,9 +58,15 @@ def forward(self,
        else:
            return self.forward_arch_param(x, arch_param=arch_param)

-    def build_arch_param(self) -> nn.Parameter:
+    def build_arch_param(self, is_random: bool = True) -> nn.Parameter:


rebase dev-1.x and move build_arch_param from mutable to mutator

pprp · 2022-08-22T00:27:38Z

mmrazor/engine/optimizers/dsnas_optimizer_wrapper.py

+
+@OPTIM_WRAPPERS.register_module()
+class DsnasOptimWrapper(OptimWrapper):
+    """Optimizer wrapper provides a common interface for updating parameters.


Please update the docstring of DsnasOptimWrapper and describe the difference between DsnasOptimWrapper and OptimWrapper.

DsnasOptimWrapper will be dropped, retain_graph=True is supposed to be implemented in mmengine

pprp · 2022-08-22T00:29:40Z

mmrazor/engine/optimizers/dsnas_optimizer_wrapper.py

+        The subclass should ensure that once :meth:`update_params` is called,
+        ``_inner_count += 1`` is automatically performed.
+
+    Examples:


Delete the Examples or add new examples for DsnasOptimWrapper.

DsnasOptimWrapper is dropped.

pprp · 2022-08-22T00:33:21Z

configs/nas/mmcls/dsnas/dsnas_supernet_8xb512_imagenet.py

+                loss_weight=1.0),
+            topk=(1, 5))),
+    mutator=dict(type='mmrazor.DiffModuleMutator'),
+    pretrain_epochs=0,


Is it normal to set pretrain_epochs to zero?

pprp · 2022-08-22T00:34:41Z

mmrazor/models/algorithms/nas/dsnas.py

+
+    Note:
+        Dsnas doesn't require retraining. It has 3 stages in searching:
+        1. `cur_epoch` < `pretrain_epochs` refers to supernet pretraining.


In the Config dsnas_shufflenet_supernet, the pretrain_epochs is set to zero.

pprp · 2022-08-22T00:42:31Z

mmrazor/engine/hooks/early_fix_arch_hook.py

+            return
+
+        model = runner.model.module
+        mutator = model.mutator


Please add type hints for mutator, because the mutator here (DiffModuleMutator) is not suitable for all kinds of mutators.

pprp · 2022-08-22T01:09:58Z

mmrazor/models/mutables/mutable_module/diff_mutable_module.py

+            Tensor: Sampled one-hot arch weights.
+        """
+        import torch.distributions as D
+        if random_sample:


Why not use gumbel_softmax directly?

pprp · 2022-08-22T01:15:09Z

mmrazor/models/mutators/module_mutator/diff_module_mutator.py

-        return nn.Parameter(torch.randn(num_choices) * 1e-3)
-
-    def prepare_from_supernet(self, supernet: nn.Module) -> None:
+        if is_random:


Please add docstring for is_random.

is_random is dropped.

sunnyxiaohu · 2022-09-26T11:39:53Z

mmrazor/models/algorithms/nas/dsnas.py

+
+        return mutable_module_resources
+
+    def update_mutator(self, cur_epoch: int) -> bool:


replace function name with need_update_mutator

sunnyxiaohu · 2022-09-26T11:49:13Z

mmrazor/models/algorithms/nas/dsnas.py

+                for k, v in self.mutator.arch_params.items():
+                    dist.broadcast(v, src=0)
+                self.fix_subnet()
+


Set a uniform indicator here to indicate whether to update the mutator, then we can directly use it later (L176 and L182)

* [feature] CONTRASTIVE REPRESENTATION DISTILLATION with dataset wrapper (#281) * init * TD: CRDLoss * complete UT * fix docstrings * fix ci * update * fix CI * DONE * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * add UT: CRD_ClsDataset * init * TODO: UT test formatting. * init * crd dataset wrapper * update docstring Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> * [Improvement] Update estimator with api revision (#277) * update estimator usage and fix bugs * refactor api of estimator & add inner check methods * fix docstrings * update search loop and config * fix lint * update unittest * decouple mmdet dependency and fix lint Co-authored-by: humu789 <humu@pjlab.org.cn> * [Fix] Fix tracer (#273) * test image_classifier_loss_calculator * fix backward tracer * update SingleStageDetectorPseudoLoss * merge * [Feature] Add Dsnas Algorithm (#226) * [tmp] Update Dsnas * [tmp] refactor arch_loss & flops_loss * Update Dsnas & MMRAZOR_EVALUATOR: 1. finalized compute_loss & handle_grads in algorithm; 2. add MMRAZOR_EVALUATOR; 3. fix bugs. * Update lr scheduler & fix a bug: 1. update param_scheduler & lr_scheduler for dsnas; 2. fix a bug of switching to finetune stage. * remove old evaluators * remove old evaluators * update param_scheduler config * merge dev-1.x into gy/estimator * add flops_loss in Dsnas using ResourcesEstimator * get resources before mutator.prepare_from_supernet * delete unness broadcast api from gml * broadcast spec_modules_resources when estimating * update early fix mechanism for Dsnas * fix merge * update units in estimator * minor change * fix data_preprocessor api * add flops_loss_coef * remove DsnasOptimWrapper * fix bn eps and data_preprocessor * fix bn weight decay bug * add betas for mutator optimizer * set diff_rank_seed=True for dsnas * fix start_factor of lr when warm up * remove .module in non-ddp mode * add GlobalAveragePoolingWithDropout * add UT for dsnas * remove unness channel adjustment for shufflenetv2 * update supernet configs * delete unness dropout * delete unness part with minor change on dsnas * minor change on the flag of search stage * update README and subnet configs * add UT for OneHotMutableOP * [Feature] Update train (#279) * support auto resume * add enable auto_scale_lr in train.py * support '--amp' option * [Fix] Fix darts metafile (#278) fix darts metafile * fix ci (#284) * fix ci for circle ci * fix bug in test_metafiles * add pr_stage_test for github ci * add multiple version * fix ut * fix lint * Temporarily skip dataset UT * update github ci * add github lint ci * install wheel * remove timm from requirements * install wheel when test on windows * fix error * fix bug * remove github windows ci * fix device error of arch_params when DsnasDDP * fix CRD dataset ut * fix scope error * rm test_cuda in workflows of github * [Doc] fix typos in en/usr_guides Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn> Co-authored-by: P.Huang <37200926+FreakieHuang@users.noreply.github.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: Yang Gao <Gary1546308416AL@gmail.com> Co-authored-by: humu789 <humu@pjlab.org.cn> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn>

* [feature] CONTRASTIVE REPRESENTATION DISTILLATION with dataset wrapper (#281) * init * TD: CRDLoss * complete UT * fix docstrings * fix ci * update * fix CI * DONE * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * add UT: CRD_ClsDataset * init * TODO: UT test formatting. * init * crd dataset wrapper * update docstring Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> * [Improvement] Update estimator with api revision (#277) * update estimator usage and fix bugs * refactor api of estimator & add inner check methods * fix docstrings * update search loop and config * fix lint * update unittest * decouple mmdet dependency and fix lint Co-authored-by: humu789 <humu@pjlab.org.cn> * [Fix] Fix tracer (#273) * test image_classifier_loss_calculator * fix backward tracer * update SingleStageDetectorPseudoLoss * merge * [Feature] Add Dsnas Algorithm (#226) * [tmp] Update Dsnas * [tmp] refactor arch_loss & flops_loss * Update Dsnas & MMRAZOR_EVALUATOR: 1. finalized compute_loss & handle_grads in algorithm; 2. add MMRAZOR_EVALUATOR; 3. fix bugs. * Update lr scheduler & fix a bug: 1. update param_scheduler & lr_scheduler for dsnas; 2. fix a bug of switching to finetune stage. * remove old evaluators * remove old evaluators * update param_scheduler config * merge dev-1.x into gy/estimator * add flops_loss in Dsnas using ResourcesEstimator * get resources before mutator.prepare_from_supernet * delete unness broadcast api from gml * broadcast spec_modules_resources when estimating * update early fix mechanism for Dsnas * fix merge * update units in estimator * minor change * fix data_preprocessor api * add flops_loss_coef * remove DsnasOptimWrapper * fix bn eps and data_preprocessor * fix bn weight decay bug * add betas for mutator optimizer * set diff_rank_seed=True for dsnas * fix start_factor of lr when warm up * remove .module in non-ddp mode * add GlobalAveragePoolingWithDropout * add UT for dsnas * remove unness channel adjustment for shufflenetv2 * update supernet configs * delete unness dropout * delete unness part with minor change on dsnas * minor change on the flag of search stage * update README and subnet configs * add UT for OneHotMutableOP * [Feature] Update train (#279) * support auto resume * add enable auto_scale_lr in train.py * support '--amp' option * [Fix] Fix darts metafile (#278) fix darts metafile * fix ci (#284) * fix ci for circle ci * fix bug in test_metafiles * add pr_stage_test for github ci * add multiple version * fix ut * fix lint * Temporarily skip dataset UT * update github ci * add github lint ci * install wheel * remove timm from requirements * install wheel when test on windows * fix error * fix bug * remove github windows ci * fix device error of arch_params when DsnasDDP * fix CRD dataset ut * fix scope error * rm test_cuda in workflows of github * [Doc] fix typos in en/usr_guides Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn> * fix bug when python=3.6 * fix lint * fix bug when test using cpu only * refine ci * fix error in ci * try ci * update repr of Channel * fix error * mv init_from_predefined_model to MutableChannelUnit * move tests * update SquentialMutableChannel * update l1 mutable channel unit * add OneShotMutableChannel * candidate_mode -> choice_mode * update docstring * change ci Co-authored-by: P.Huang <37200926+FreakieHuang@users.noreply.github.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: Yang Gao <Gary1546308416AL@gmail.com> Co-authored-by: humu789 <humu@pjlab.org.cn> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn>

* add ChannelGroup (#250) * rebase new dev-1.x * modification for adding config_template * add docstring to channel_group.py * add docstring to mutable_channel_group.py * rm channel_group_cfg from Graph2ChannelGroups * change choice type of SequentialChannelGroup from float to int * add a warning about group-wise conv * restore __init__ of dynamic op * in_channel_mutable -> mutable_in_channel * rm abstractproperty * add a comment about VT * rm registry for ChannelGroup * MUTABLECHANNELGROUP -> ChannelGroupType * refine docstring of IndexDict * update docstring * update docstring * is_prunable -> is_mutable * update docstring * fix error in pre-commit * update unittest * add return type * unify init_xxx apit * add unitest about init of MutableChannelGroup * update according to reviews * sequential_channel_group -> sequential_mutable_channel_group Co-authored-by: liukai <liukai@pjlab.org.cn> * Add BaseChannelMutator and refactor Autoslim (#289) * add BaseChannelMutator * add autoslim * tmp * make SequentialMutableChannelGroup accpeted both of num and ratio as choice. and supports divisior * update OneShotMutableChannelGroup * pass supernet training of autoslim * refine autoslim * fix bug in OneShotMutableChannelGroup * refactor make_divisible * fix spell error: channl -> channel * init_using_backward_tracer -> init_from_backward_tracer init_from_fx_tracer -> init_from_fx_tracer * refine SequentialMutableChannelGroup * let mutator support models with dynamicop * support define search space in model * tracer_cfg -> parse_cfg * refine * using -> from * update docstring * update docstring Co-authored-by: liukai <liukai@pjlab.org.cn> * refactor slimmable and add l1-norm (#291) * refactor slimmable and add l1-norm * make l1-norm support convnd * update get_channel_groups * add l1-norm_resnet34_8xb32_in1k.py * add pretrained to resnet34-l1 * remove old channel mutator * BaseChannelMutator -> ChannelMutator * update according to reviews * add readme to l1-norm * MBV2_slimmable -> MBV2_slimmable_config Co-authored-by: liukai <liukai@pjlab.org.cn> * Clean old codes. (#296) * remove old dynamic ops * move dynamic ops * clean old mutable_channels * rm OneShotMutableChannel * rm MutableChannel * refine * refine * use SquentialMutableChannel to replace OneshotMutableChannel * refactor dynamicops folder * let SquentialMutableChannel support float Co-authored-by: liukai <liukai@pjlab.org.cn> * Add channel-flow (#301) * base_channel_mutator -> channel_mutator * init * update docstring * allow omitting redundant configs for channel * add register_mutable_channel_to_a_module to MutableChannelContainer * update according to reviews 1 * update according to reviews 2 * update according to reviews 3 * remove old docstring * fix error * using->from * update according to reviews * support self-define input channel number * update docstring * chanenl -> channel_elem Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: jacky <jacky@xx.com> * Rename: ChannelGroup -> ChannelUnit (#302) * refine repr of MutableChannelGroup * rename folder name * ChannelGroup -> ChannelUnit * filename in units folder * channel_group -> channel_unit * groups -> units * group -> unit * update * get_mutable_channel_groups -> get_mutable_channel_units * fix bug * refine docstring * fix ci * fix bug in tracer Co-authored-by: liukai <liukai@pjlab.org.cn> * Merge dev-1.x to pruning (#311) * [feature] CONTRASTIVE REPRESENTATION DISTILLATION with dataset wrapper (#281) * init * TD: CRDLoss * complete UT * fix docstrings * fix ci * update * fix CI * DONE * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * add UT: CRD_ClsDataset * init * TODO: UT test formatting. * init * crd dataset wrapper * update docstring Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> * [Improvement] Update estimator with api revision (#277) * update estimator usage and fix bugs * refactor api of estimator & add inner check methods * fix docstrings * update search loop and config * fix lint * update unittest * decouple mmdet dependency and fix lint Co-authored-by: humu789 <humu@pjlab.org.cn> * [Fix] Fix tracer (#273) * test image_classifier_loss_calculator * fix backward tracer * update SingleStageDetectorPseudoLoss * merge * [Feature] Add Dsnas Algorithm (#226) * [tmp] Update Dsnas * [tmp] refactor arch_loss & flops_loss * Update Dsnas & MMRAZOR_EVALUATOR: 1. finalized compute_loss & handle_grads in algorithm; 2. add MMRAZOR_EVALUATOR; 3. fix bugs. * Update lr scheduler & fix a bug: 1. update param_scheduler & lr_scheduler for dsnas; 2. fix a bug of switching to finetune stage. * remove old evaluators * remove old evaluators * update param_scheduler config * merge dev-1.x into gy/estimator * add flops_loss in Dsnas using ResourcesEstimator * get resources before mutator.prepare_from_supernet * delete unness broadcast api from gml * broadcast spec_modules_resources when estimating * update early fix mechanism for Dsnas * fix merge * update units in estimator * minor change * fix data_preprocessor api * add flops_loss_coef * remove DsnasOptimWrapper * fix bn eps and data_preprocessor * fix bn weight decay bug * add betas for mutator optimizer * set diff_rank_seed=True for dsnas * fix start_factor of lr when warm up * remove .module in non-ddp mode * add GlobalAveragePoolingWithDropout * add UT for dsnas * remove unness channel adjustment for shufflenetv2 * update supernet configs * delete unness dropout * delete unness part with minor change on dsnas * minor change on the flag of search stage * update README and subnet configs * add UT for OneHotMutableOP * [Feature] Update train (#279) * support auto resume * add enable auto_scale_lr in train.py * support '--amp' option * [Fix] Fix darts metafile (#278) fix darts metafile * fix ci (#284) * fix ci for circle ci * fix bug in test_metafiles * add pr_stage_test for github ci * add multiple version * fix ut * fix lint * Temporarily skip dataset UT * update github ci * add github lint ci * install wheel * remove timm from requirements * install wheel when test on windows * fix error * fix bug * remove github windows ci * fix device error of arch_params when DsnasDDP * fix CRD dataset ut * fix scope error * rm test_cuda in workflows of github * [Doc] fix typos in en/usr_guides Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn> Co-authored-by: P.Huang <37200926+FreakieHuang@users.noreply.github.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: Yang Gao <Gary1546308416AL@gmail.com> Co-authored-by: humu789 <humu@pjlab.org.cn> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn> * Refine pruning branch (#307) * [feature] CONTRASTIVE REPRESENTATION DISTILLATION with dataset wrapper (#281) * init * TD: CRDLoss * complete UT * fix docstrings * fix ci * update * fix CI * DONE * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * maintain CRD dataset unique funcs as a mixin * add UT: CRD_ClsDataset * init * TODO: UT test formatting. * init * crd dataset wrapper * update docstring Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> * [Improvement] Update estimator with api revision (#277) * update estimator usage and fix bugs * refactor api of estimator & add inner check methods * fix docstrings * update search loop and config * fix lint * update unittest * decouple mmdet dependency and fix lint Co-authored-by: humu789 <humu@pjlab.org.cn> * [Fix] Fix tracer (#273) * test image_classifier_loss_calculator * fix backward tracer * update SingleStageDetectorPseudoLoss * merge * [Feature] Add Dsnas Algorithm (#226) * [tmp] Update Dsnas * [tmp] refactor arch_loss & flops_loss * Update Dsnas & MMRAZOR_EVALUATOR: 1. finalized compute_loss & handle_grads in algorithm; 2. add MMRAZOR_EVALUATOR; 3. fix bugs. * Update lr scheduler & fix a bug: 1. update param_scheduler & lr_scheduler for dsnas; 2. fix a bug of switching to finetune stage. * remove old evaluators * remove old evaluators * update param_scheduler config * merge dev-1.x into gy/estimator * add flops_loss in Dsnas using ResourcesEstimator * get resources before mutator.prepare_from_supernet * delete unness broadcast api from gml * broadcast spec_modules_resources when estimating * update early fix mechanism for Dsnas * fix merge * update units in estimator * minor change * fix data_preprocessor api * add flops_loss_coef * remove DsnasOptimWrapper * fix bn eps and data_preprocessor * fix bn weight decay bug * add betas for mutator optimizer * set diff_rank_seed=True for dsnas * fix start_factor of lr when warm up * remove .module in non-ddp mode * add GlobalAveragePoolingWithDropout * add UT for dsnas * remove unness channel adjustment for shufflenetv2 * update supernet configs * delete unness dropout * delete unness part with minor change on dsnas * minor change on the flag of search stage * update README and subnet configs * add UT for OneHotMutableOP * [Feature] Update train (#279) * support auto resume * add enable auto_scale_lr in train.py * support '--amp' option * [Fix] Fix darts metafile (#278) fix darts metafile * fix ci (#284) * fix ci for circle ci * fix bug in test_metafiles * add pr_stage_test for github ci * add multiple version * fix ut * fix lint * Temporarily skip dataset UT * update github ci * add github lint ci * install wheel * remove timm from requirements * install wheel when test on windows * fix error * fix bug * remove github windows ci * fix device error of arch_params when DsnasDDP * fix CRD dataset ut * fix scope error * rm test_cuda in workflows of github * [Doc] fix typos in en/usr_guides Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn> * fix bug when python=3.6 * fix lint * fix bug when test using cpu only * refine ci * fix error in ci * try ci * update repr of Channel * fix error * mv init_from_predefined_model to MutableChannelUnit * move tests * update SquentialMutableChannel * update l1 mutable channel unit * add OneShotMutableChannel * candidate_mode -> choice_mode * update docstring * change ci Co-authored-by: P.Huang <37200926+FreakieHuang@users.noreply.github.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: Yang Gao <Gary1546308416AL@gmail.com> Co-authored-by: humu789 <humu@pjlab.org.cn> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn> Co-authored-by: liukai <liukai@pjlab.org.cn> Co-authored-by: jacky <jacky@xx.com> Co-authored-by: P.Huang <37200926+FreakieHuang@users.noreply.github.com> Co-authored-by: huangpengsheng <huangpengsheng@sensetime.com> Co-authored-by: Yang Gao <Gary1546308416AL@gmail.com> Co-authored-by: humu789 <humu@pjlab.org.cn> Co-authored-by: whcao <41630003+HIT-cwh@users.noreply.github.com> Co-authored-by: pppppM <gjf_mail@126.com> Co-authored-by: gaoyang07 <1546308416@qq.com> Co-authored-by: SheffieldCao <1751899@tongji.edu.cn>

* add cls trt benchmark * add benchmark.md * fix ~ * resolve comments * merge master * caption Det * resolve comments * conditions -> settings

Co-authored-by: 程建刚 <chengjiangang@mail206.com>

gaoyang07 added 3 commits August 1, 2022 16:09

[tmp] Update Dsnas

6fad48c

[tmp] refactor arch_loss & flops_loss

6ded2d4

Update Dsnas & MMRAZOR_EVALUATOR:

1932d82

1. finalized compute_loss & handle_grads in algorithm; 2. add MMRAZOR_EVALUATOR; 3. fix bugs.

gaoyang07 requested review from pppppM and sunnyxiaohu August 10, 2022 09:55

Merge branch 'dev-1.x' into gy/dsnas

be98e8b

Update lr scheduler & fix a bug:

b1f5c7d

1. update param_scheduler & lr_scheduler for dsnas; 2. fix a bug of switching to finetune stage.

sunnyxiaohu reviewed Aug 12, 2022

View reviewed changes

gaoyang07 changed the title ~~[WIP] Dsnas and New evaluator~~ [WIP] [Feature] Add Dsnas Algorithm Aug 15, 2022

gaoyang07 added 4 commits August 16, 2022 14:25

remove old evaluators

8f8beef

Merge branch 'dev-1.x' of github.com:open-mmlab/mmrazor into gy/dsnas

6f4e6e8

remove old evaluators

3deb833

update param_scheduler config

2247bdd

pprp reviewed Aug 22, 2022

View reviewed changes

gaoyang07 added 15 commits August 23, 2022 17:43

merge dev-1.x into gy/estimator

8d8d1b8

add flops_loss in Dsnas using ResourcesEstimator

c2dbcaf

get resources before mutator.prepare_from_supernet

bc813f3

delete unness broadcast api from gml

ce87f89

broadcast spec_modules_resources when estimating

73d9c3b

update early fix mechanism for Dsnas

5de5bc9

merge dev-1.x into gy/dsnas

1e4014d

fix merge

e98043a

update units in estimator

2fbdd01

minor change

d676d93

merge dev-1.x into gy/dsnas

420dcac

fix data_preprocessor api

d6a401b

add flops_loss_coef

aa2e0f2

remove DsnasOptimWrapper

08964ce

fix bn eps and data_preprocessor

32baf69

gaoyang07 force-pushed the gy/dsnas branch from a3f7017 to 32baf69 Compare August 31, 2022 08:16

gaoyang07 added 10 commits September 7, 2022 00:13

fix bn weight decay bug

22cf3ed

add betas for mutator optimizer

e0bafe2

set diff_rank_seed=True for dsnas

84b5367

fix start_factor of lr when warm up

eef0514

remove .module in non-ddp mode

7258926

add GlobalAveragePoolingWithDropout

2dfdc79

add UT for dsnas

79d69ef

remove unness channel adjustment for shufflenetv2

b511403

update supernet configs

91ef7d8

delete unness dropout

cfccb5e

gaoyang07 changed the title ~~[WIP] [Feature] Add Dsnas Algorithm~~ [Feature] Add Dsnas Algorithm Sep 26, 2022

delete unness part with minor change on dsnas

6794188

sunnyxiaohu reviewed Sep 28, 2022

View reviewed changes

gaoyang07 added 4 commits September 29, 2022 12:54

merge dev-1.x into gy/dsnas

bbc93ff

minor change on the flag of search stage

3a6b696

update README and subnet configs

6df18d1

add UT for OneHotMutableOP

a77d894

sunnyxiaohu self-requested a review September 29, 2022 08:46

sunnyxiaohu merged commit 8d603d9 into open-mmlab:dev-1.x Sep 29, 2022

humu789 pushed a commit to humu789/mmrazor that referenced this pull request Feb 13, 2023

[Docs] Trt speed docs (open-mmlab#226)

ad3cd5a

* add cls trt benchmark * add benchmark.md * fix ~ * resolve comments * merge master * caption Det * resolve comments * conditions -> settings

humu789 pushed a commit to humu789/mmrazor that referenced this pull request Feb 13, 2023

set the default value of MMDEPLOY_CODEBASES to all (open-mmlab#226)

df4e9e6

Co-authored-by: 程建刚 <chengjiangang@mail206.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add Dsnas Algorithm #226

[Feature] Add Dsnas Algorithm #226

gaoyang07 commented Aug 10, 2022 •

edited

Loading

codecov bot commented Aug 11, 2022 •

edited

Loading

sunnyxiaohu Aug 10, 2022

pprp Aug 22, 2022

gaoyang07 Aug 22, 2022

pprp Aug 22, 2022

gaoyang07 Sep 28, 2022

pprp Aug 22, 2022

pprp Aug 22, 2022

pprp Aug 22, 2022

pprp Aug 22, 2022

pprp Aug 22, 2022

gaoyang07 Sep 28, 2022

sunnyxiaohu Sep 26, 2022

gaoyang07 Sep 29, 2022

sunnyxiaohu Sep 26, 2022

gaoyang07 Sep 29, 2022


		return mutable_module_resources

		def update_mutator(self, cur_epoch: int) -> bool:

[Feature] Add Dsnas Algorithm #226

[Feature] Add Dsnas Algorithm #226

Conversation

gaoyang07 commented Aug 10, 2022 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented Aug 11, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaoyang07 commented Aug 10, 2022 •

edited

Loading

codecov bot commented Aug 11, 2022 •

edited

Loading