[Enhance] Support dynamic input shape for ViT-based algorithms

[mzr1996]

Motivation

Currently, our ViT-based algorithms need to specify the image shape. But in some downstream tasks, they need input images with various shapes.

Modification

Refactor these ViT-based algorithms to support dynamic input shape as far as possible.

• ✔️ VisionTransformer
• ✔️ SwinTransformer
• ✔️ DeiT
• ❌ MLP-Mixer (The structure doesn't support dynamic input shape)
• ✔️ T2T-ViT

BC-breaking (Optional)

The input_resolution and auto_pad argument are deprecated in the ShiftWindowMSA. And the auto_pad argument is removed from SwinTransformer.

Use cases (Optional)

>>> from mmcls.models import VisionTransformer
>>> import torch
>>> model = VisionTransformer(arch="base")
>>> inputs = torch.rand(1, 3, 256, 323)  # The input shape is no-limit.
>>> patch_embed, cls_token = model(inputs)[0]
>>> print(patch_embed.shape)
torch.Size([1, 768, 16, 21])
>>> print(cls_token.shape)
torch.Size([1, 768])

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
• CLA has been signed and all committers have signed the CLA in this PR.

[codecov]

Codecov Report

Merging #706 (d0d81e6) into dev (28e3bc8) will decrease coverage by 0.27%.
The diff coverage is 96.63%.

@@            Coverage Diff             @@
##              dev     #706      +/-   ##
==========================================
- Coverage   85.22%   84.94%   -0.28%     
==========================================
  Files         121      121              
  Lines        7432     7548     +116     
  Branches     1278     1303      +25     
==========================================
+ Hits         6334     6412      +78     
- Misses        911      944      +33     
- Partials      187      192       +5     

Flag	Coverage Δ
unittests	84.87% <96.63%> (-0.31%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/models/backbones/swin_transformer.py	91.97% <87.80%> (-0.03%)	⬇️
mmcls/models/backbones/t2t_vit.py	95.12% <96.15%> (+0.59%)	⬆️
mmcls/models/backbones/deit.py	97.82% <100.00%> (+0.32%)	⬆️
mmcls/models/backbones/mlp_mixer.py	96.77% <100.00%> (+1.16%)	⬆️
mmcls/models/backbones/vision_transformer.py	96.37% <100.00%> (-0.75%)	⬇️
mmcls/models/utils/__init__.py	100.00% <100.00%> (ø)
mmcls/models/utils/attention.py	100.00% <100.00%> (ø)
mmcls/models/utils/embed.py	80.00% <100.00%> (-0.59%)	⬇️
mmcls/datasets/dataset_wrappers.py	70.58% <0.00%> (-21.17%)	⬇️
... and 8 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 28e3bc8...d0d81e6. Read the comment docs.

[Ezra-Yu]

please add DeprecationWarning in PatchEmbed and PatchMerge.

[Ezra-Yu]

LGTM.

[ZwwWayne]

This introduces BC-breaking in mmselfsup 0.6.0.

[ShangWeiKuo]

Excuse me @mzr1996 . Could you please explain how you did to make ViT-based algorithms to support dynamic input shape?