[WIP] Add Swin Transformer #511

zeliu98 · 2021-04-23T17:21:03Z

No description provided.

CLAassistant · 2021-04-23T17:21:09Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.

✅ xvjiarui
✅ Junjun2016
✅ sennnnn
❌ zeliu98
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

xvjiarui · 2021-06-25T15:57:39Z

mmcv_custom/checkpoint.py

@@ -0,0 +1,496 @@
+# Copyright (c) Open-MMLab. All rights reserved.


We may remove this file and directory.

xvjiarui · 2021-06-25T15:58:38Z

mmseg/models/backbones/swin_transformer.py

+                init_cfg=None,
+            )


Suggested change

init_cfg=None,

)

init_cfg=None)

xvjiarui · 2021-06-25T16:01:52Z

mmseg/models/backbones/swin_transformer.py

+        stride = to_2tuple(stride)
+        padding = to_2tuple(padding)
+        dilation = to_2tuple(dilation)
+        self.sampler = nn.Unfold(kernel_size, dilation, padding, stride)


Check if https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation/blob/87e6f90577435c94f3e92c7db1d36edc234d91f6/mmseg/models/backbones/swin_transformer.py#L283-L286 is also correctly handled.

The padding may need to calculate by users.

xvjiarui · 2021-06-25T16:05:20Z

mmseg/models/backbones/swin_transformer.py

+
+
+@ATTENTION.register_module()
+class ShiftWindowMSA(BaseModule):


Missing docstring.

xvjiarui · 2021-06-25T16:06:04Z

mmseg/models/backbones/swin_transformer.py

+class ShiftWindowMSA(BaseModule):
+
+    def __init__(self,
+                 input_resolution,


In our setting, the input size may change during infer or training. We should determine input size when initializing the module.

xvjiarui · 2021-06-25T16:06:26Z

mmseg/models/backbones/swin_transformer.py

+        return windows
+
+
+class SwinBlock(BaseModule):


Missing docstring.

xvjiarui · 2021-06-25T16:06:48Z

mmseg/models/backbones/swin_transformer.py

+class SwinBlock(BaseModule):
+
+    def __init__(self,
+                 input_resolution,


Similarly, input_size should be unknown.

xvjiarui · 2021-06-25T16:08:23Z

mmseg/models/backbones/swin_transformer.py

+    def forward(self, query):
+        for block in self.blocks:
+            query = block(query)
+
+        if self.downsample:
+            query = self.downsample(query)
+        return query


We should have something like
https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation/blob/87e6f90577435c94f3e92c7db1d36edc234d91f6/mmseg/models/backbones/swin_transformer.py#L362

I suggest pack (H, W) into hw_shape, and forward it also.

Actually, the H, W are wrapped in the attributes of PatchMerging. H, W = self.output_resolution.

codecov · 2021-06-26T07:23:03Z

Codecov Report

Merging #511 (3ac0547) into master (98067be) will decrease coverage by 0.62%.
The diff coverage is 72.06%.

❗ Current head 3ac0547 differs from pull request most recent head 4a00406. Consider uploading reports for the commit 4a00406 to get more accurate results

@@            Coverage Diff             @@
##           master     #511      +/-   ##
==========================================
- Coverage   85.77%   85.14%   -0.63%     
==========================================
  Files         103      105       +2     
  Lines        5307     5668     +361     
  Branches      857      923      +66     
==========================================
+ Hits         4552     4826     +274     
- Misses        583      663      +80     
- Partials      172      179       +7

Flag	Coverage Δ
unittests	`85.14% <72.06%> (-0.63%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/models/utils/ckpt_convert.py	`4.28% <4.28%> (ø)`
mmseg/models/utils/embed.py	`80.55% <80.55%> (ø)`
mmseg/models/backbones/swin.py	`86.89% <86.89%> (ø)`
mmseg/models/backbones/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/backbones/vit.py	`84.84% <100.00%> (-1.03%)`	⬇️
mmseg/models/utils/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/necks/multilevel_neck.py	`100.00% <0.00%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98067be...4a00406. Read the comment docs.

xvjiarui · 2021-06-28T22:59:45Z

mmseg/models/backbones/swin.py

+class ShiftWindowMSA(BaseModule):
+
+    def __init__(self,
+                 input_resolution,


We shouldn't have input_resolution arg.
Reference: https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation/blob/87e6f90577435c94f3e92c7db1d36edc234d91f6/mmseg/models/backbones/swin_transformer.py#L156

xvjiarui · 2021-06-28T23:01:02Z

mmseg/models/utils/embed.py

@@ -0,0 +1,91 @@
+import torch


Modified from xxx.

xvjiarui · 2021-06-29T03:46:25Z

mmseg/models/backbones/swin.py

+        else:
+            self.downsample = None
+
+    def forward(self, x, H, W):


We may pack H, W into hw_shape as a tuple.
Reference https://github.com/facebookresearch/SlowFast/blob/2090f2918ac1ce890fdacd8fda2e590a46d5c734/slowfast/models/video_model_builder.py#L1002

We also need a docstring for this.

xvjiarui · 2021-06-29T03:47:09Z

mmseg/models/backbones/swin.py

+        if self.downsample:
+            stage_out = x
+            x = self.downsample(x, H, W)
+            DH, DW = (H + 1) // 2, (W + 1) // 2
+            return stage_out, H, W, x, DH, DW
+        else:
+            return x, H, W, x, H, W


The output should be x, hw_shape.
We don't need the previous hw_shape.

xvjiarui · 2021-06-29T18:10:34Z

mmseg/models/backbones/swin.py

+                 stride=None,
+                 padding=0,
+                 dilation=1,


Remove these args. Make stride=kernel_size.

2. Correct weight convert function; 3. Fix the pad of Patch Merging;

xvjiarui · 2021-06-30T05:35:59Z

mmseg/models/backbones/swin.py

+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size=2,


We may remove this.

xvjiarui · 2021-06-30T05:39:53Z

mmseg/models/backbones/swin.py

+                 mlp_ratio=4,
+                 depths=(2, 2, 6, 2),
+                 num_heads=(3, 6, 12, 24),
+                 strides=(None, None, None, None),


Why the default is none?

xvjiarui · 2021-06-30T05:44:30Z

mmseg/models/backbones/swin.py

+        num_heads (int): Parallel attention heads.
+        feedforward_channels (int): The hidden dimension for FFNs.
+        depth (int): The number of blocks in this stage.
+        kernel_size (int): The kernel_size of patch merging.


We may also remove this. Use stride only.

xvjiarui · 2021-06-30T05:44:45Z

mmseg/models/backbones/swin.py

+        padding (int): The padding length of patch merging.
+        dilation (int): The dilation rate of kernel of patch merging.


Not needed.

xvjiarui · 2021-06-30T05:45:01Z

mmseg/models/backbones/swin.py

+                 kernel_size,
+                 stride,
+                 padding,
+                 dilation,


Keep stride only.

mmseg/models/backbones/swin.py

2. Fix some pad bug; 3. Modify config to adapt new swin implementation;

xvjiarui · 2021-06-30T22:12:31Z

mmseg/models/backbones/swin.py

+        paddings (tuple[int], optional): The patch merging or patch
+            embedding padding length of each Swin Transformer stage.
+            Default: (0, 0, 0, 0).
+        dilations (tuple[int], optional): The patch merging or patch
+            embedding kernel dilation rate of each Swin Transformer stage.
+            Default: (1, 1, 1, 1).


These are no longer needed.

xvjiarui · 2021-06-30T22:13:34Z

mmseg/models/backbones/swin.py

+            if downsample:
+                in_channels = in_channels * 2


Move to the definition of downample.

2. Modify pth url which keep meta attribute;

…into swin_transformer

Add ViT link

* add Swin Transformer * add Swin Transformer * fixed import * Add some swin training settings. * Fix some filename error. * Fix attribute name: pretrain -> pretrained * Upload mmcls implementation of swin transformer. * Refactor Swin Transformer to follow mmcls style. * Refactor init_weigths of swin_transformer.py * Fix lint * Match inference precision * Add some comments * Add swin_convert to load official style ckpt * Remove arg: auto_pad * 1. Complete comments for each block; 2. Correct weight convert function; 3. Fix the pad of Patch Merging; * Clean function args. * Fix vit unit test. * 1. Add swin transformer unit tests; 2. Fix some pad bug; 3. Modify config to adapt new swin implementation; * Modify config arg * Update readme.md of swin * Fix config arg error and Add some swin benchmark msg. * Add MeM and ms test content for readme.md of swin transformer. * Fix doc string of swin module * 1. Register swin transformer to model list; 2. Modify pth url which keep meta attribute; * Update swin.py * Merge config settings. * Modify config style. * Update README.md Add ViT link * Modify main readme.md Co-authored-by: Jiarui XU <xvjiarui0826@gmail.com> Co-authored-by: sennnnn <201730271412@mail.scut.edu.cn> Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>

…ful). (open-mmlab#511) * Removing `autocast` for `35-25% speedup`. * iQuality * Adding a slow test. * Fixing mps noise generation. * Raising error on wrong device, instead of just casting on behalf of user. * Quality. * fix merge Co-authored-by: Nouamane Tazi <nouamane98@gmail.com>

* resolve comments * update changelog * add test_batch * add testing for `test_batch` * fix mmcv version * add test_batch * add testing for `test_batch` * enlarge test_input to pass unittest * update names * update changelog & faq * update name

zeliu98 added 2 commits April 24, 2021 01:06

add Swin Transformer

dc302c5

add Swin Transformer

29266c0

zeliu98 changed the title ~~add Swin transformer~~ add Swin Transformer Apr 23, 2021

clownrat6 mentioned this pull request May 5, 2021

SETR and SWIN transformers #528

Closed

xvjiarui and others added 10 commits May 23, 2021 09:36

Merge branch 'master' into swin_transformer

50430f0

fixed import

ac1e66d

Add some swin training settings.

844deeb

Fix some filename error.

4df970f

Fix attribute name: pretrain -> pretrained

52c7c58

Merge branch 'master' into swin_transformer

b888d15

Merge branch 'master' into swin_transformer

7b54553

Upload mmcls implementation of swin transformer.

f98e864

Refactor Swin Transformer to follow mmcls style.

2a01b13

Merge Master

457ce05

xvjiarui reviewed Jun 25, 2021

View reviewed changes

Refactor init_weigths of swin_transformer.py

6ffde48

Fix lint

d4136db

clownrat6 changed the title ~~add Swin Transformer~~ [WIP] add Swin Transformer Jun 28, 2021

clownrat6 changed the title ~~[WIP] add Swin Transformer~~ [WIP] Add Swin Transformer Jun 28, 2021

Match inference precision

e5d914c

xvjiarui reviewed Jun 28, 2021

View reviewed changes

mmseg/models/utils/embed.py Outdated

@@ -0,0 +1,91 @@

import torch

Copy link

Collaborator

xvjiarui Jun 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified from xxx.

Add some comments

5a6d6a6

xvjiarui reviewed Jun 29, 2021

View reviewed changes

1. Complete comments for each block;

773d4e3

2. Correct weight convert function; 3. Fix the pad of Patch Merging;

xvjiarui reviewed Jun 30, 2021

View reviewed changes

mmseg/models/backbones/swin.py Outdated

def __init__(self,

in_channels,

out_channels,

kernel_size=2,

Copy link

Collaborator

xvjiarui Jun 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may remove this.

xvjiarui reviewed Jun 30, 2021

View reviewed changes

mmseg/models/backbones/swin.py Show resolved Hide resolved

sennnnn added 6 commits June 30, 2021 13:58

Clean function args.

3aba6c2

Fix vit unit test.

36e346d

1. Add swin transformer unit tests;

631dd27

2. Fix some pad bug; 3. Modify config to adapt new swin implementation;

Modify config arg

16256ed

Update readme.md of swin

07c94a3

Fix config arg error and Add some swin benchmark msg.

519f11e

xvjiarui reviewed Jun 30, 2021

View reviewed changes

sennnnn added 2 commits July 1, 2021 11:36

Add MeM and ms test content for readme.md of swin transformer.

89076ae

Fix doc string of swin module

ef1de28

xvjiarui approved these changes Jul 1, 2021

View reviewed changes

sennnnn and others added 7 commits July 1, 2021 14:30

1. Register swin transformer to model list;

d024154

2. Modify pth url which keep meta attribute;

Update swin.py

34f2507

Merge config settings.

f27fa3b

Merge branch 'swin_transformer' of github.com:zeliu98/mmsegmentation …

5ba227d

…into swin_transformer

Modify config style.

1de62fd

Update README.md

3ac0547

Add ViT link

Modify main readme.md

4a00406

Junjun2016 merged commit b6c7c77 into open-mmlab:master Jul 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add Swin Transformer #511

[WIP] Add Swin Transformer #511

zeliu98 commented Apr 23, 2021

CLAassistant commented Apr 23, 2021 •

edited

Loading

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

clownrat6 Jun 26, 2021

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

xvjiarui Jun 25, 2021

clownrat6 Jun 26, 2021

codecov bot commented Jun 26, 2021 •

edited

Loading

xvjiarui Jun 28, 2021

xvjiarui Jun 28, 2021

xvjiarui Jun 29, 2021

xvjiarui Jun 29, 2021

xvjiarui Jun 29, 2021

xvjiarui Jun 29, 2021

xvjiarui Jun 30, 2021

xvjiarui Jun 30, 2021

xvjiarui Jun 30, 2021

xvjiarui Jun 30, 2021

xvjiarui Jun 30, 2021

xvjiarui Jun 30, 2021

xvjiarui Jun 30, 2021

		@@ -0,0 +1,496 @@
		# Copyright (c) Open-MMLab. All rights reserved.



		@ATTENTION.register_module()
		class ShiftWindowMSA(BaseModule):

		padding (int): The padding length of patch merging.
		dilation (int): The dilation rate of kernel of patch merging.

[WIP] Add Swin Transformer #511

[WIP] Add Swin Transformer #511

Conversation

zeliu98 commented Apr 23, 2021

CLAassistant commented Apr 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 26, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Apr 23, 2021 •

edited

Loading

codecov bot commented Jun 26, 2021 •

edited

Loading