[Feature] Add the frozen function for Swin Transformer model #574
Conversation
|
Thanks for your contribution, we will review it soon. |
|
|
| if isinstance(m, nn.modules.LayerNorm): | ||
| m.eval() |
There was a problem hiding this comment.
Does LayerNorm need to turn to evaluation mode? Unlike BatchNorm, LayerNorm should have the same behavior in both training mode and eval mode.
There was a problem hiding this comment.
It also puzzles me. I write this reference https://github.com/open-mmlab/mmclassification/blob/9d9dce69ad1d7911c87c69e30bce4a928f4dcd94/mmcls/models/backbones/resnet.py#L632 .
There was a problem hiding this comment.
No, we should remove the LayerNorm instead of _BatchNorm, because eval has an effect on BatchNorm only.
Although SwinTranformer doesn't use BatchNorm, we support using norm_cfg to replace norm type.
Codecov Report
@@ Coverage Diff @@
## master #574 +/- ##
==========================================
+ Coverage 79.75% 79.83% +0.07%
==========================================
Files 107 107
Lines 6120 6144 +24
Branches 1046 1056 +10
==========================================
+ Hits 4881 4905 +24
Misses 1105 1105
Partials 134 134
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
| @@ -291,6 +291,8 @@ def __init__(self, | |||
| use_abs_pos_embed=False, | |||
| auto_pad=False, | |||
| with_cp=False, | |||
| frozen=False, | |||
There was a problem hiding this comment.
Refers to ResNet , it's better to unify the API and use frozen_stages (int) to support freezing before a specific stage.
| if isinstance(m, nn.modules.LayerNorm): | ||
| m.eval() |
There was a problem hiding this comment.
No, we should remove the LayerNorm instead of _BatchNorm, because eval has an effect on BatchNorm only.
Although SwinTranformer doesn't use BatchNorm, we support using norm_cfg to replace norm type.
mzr1996
changed the title
…mlab#574) * Add the frozen function for Swin Transformer model * add frozen parameter for swin transformer model * add norm_eval parameter * Delete =11.1 * Delete =418,driver * delete _BatchNorm * remove LayerNorm , add _BatchNorm * unifying the style of frozen function refer ResNet * Improve docs and add unit tests. Co-authored-by: cxiang26 <cq.xiang@foxmail.com> Co-authored-by: mzr1996 <mzr1996@163.com>
Motivation
Adding feature request. It provides frozen function for swin transformer by adding add frozen(bool) parameter .
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
Checklist
Before PR:
After PR: