[Enhance] Support training on IPU and add fine-tuning configs of ViT. #723

HuDi2018 · 2022-03-04T08:08:47Z

need use a PR in mmcv
see and run tools/ipu_train_example.sh for details.

Motivation

run mmclassification ViT training on IPU

Modification

add some arguments parse in train.py and test.py to make a preparation for IPU training/inference

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

bash tools/ipu_train_example.sh

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

CLAassistant · 2022-03-04T08:08:52Z

All committers have signed the CLA.

codecov · 2022-03-04T08:15:19Z

Codecov Report

Merging #723 (be58934) into dev (02c8f82) will decrease coverage by 0.47%.
The diff coverage is 24.13%.

@@            Coverage Diff             @@
##              dev     #723      +/-   ##
==========================================
- Coverage   86.95%   86.47%   -0.48%     
==========================================
  Files         127      128       +1     
  Lines        8071     8281     +210     
  Branches     1390     1431      +41     
==========================================
+ Hits         7018     7161     +143     
- Misses        848      905      +57     
- Partials      205      215      +10

Flag	Coverage Δ
unittests	`86.41% <24.13%> (-0.46%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/apis/train.py	`15.90% <0.00%> (-2.52%)`	⬇️
mmcls/datasets/builder.py	`85.71% <40.00%> (-5.34%)`	⬇️
mmcls/datasets/pipelines/formatting.py	`43.13% <40.00%> (-0.35%)`	⬇️
mmcls/models/losses/accuracy.py	`91.89% <100.00%> (+0.11%)`	⬆️
mmcls/models/utils/embed.py	`65.33% <0.00%> (-14.67%)`	⬇️
mmcls/utils/setup_env.py	`95.45% <0.00%> (-4.55%)`	⬇️
mmcls/models/backbones/swin_transformer.py	`87.92% <0.00%> (-4.06%)`	⬇️
mmcls/apis/test.py	`23.93% <0.00%> (ø)`
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02c8f82...be58934. Read the comment docs.

Ezra-Yu · 2022-03-09T08:18:21Z

please make sure that all the commitors have signed the CLA. If the user is untracked, you can use the git-rebase to modify the git message

mzr1996

Please rebase to the latest dev branch.

configs/vision_transformer/vit-base-p16_ft-4xb510_in1k-224_ipu_half.py

.gitignore

configs/vision_transformer/vit-base-p16_ft-4xb544_in1k-224_ipu.py

hdDebug.py

configs/vision_transformer/vit-base-p16_ft-4xb544_in1k-224_ipu.py

mmcls/models/losses/accuracy.py

tools/ipu_train_example.sh

remove myconfigs in gitignore remove imagenet_bs4096_AdamW.py in config remove hdDebug.py place toHalf before collect move ipu_train_example.sh to demo

mzr1996

LGTM

* [Enhance] Add extra dataloader settings in configs. (open-mmlab#752) * Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings in the `data` field to specify different arguments. * Fix bug * Fix bug * [Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762) * [Feature] Support resize relative position embedding in `SwinTransformer`. (open-mmlab#749) * [Feature]: Add resize rel pos embed * [Refactor]: Create a separated resize_rel_pos_bias_table func * [Refactor]: Refactor rel pos embed bias * [Refactor]: Move interpolate into func * Remove index buffer only when window_size changes Co-authored-by: mzr1996 <mzr1996@163.com> * [Feature] Add PoolFormer backbone and checkpoints. (open-mmlab#746) * add PoolFormer * fix some typos in PoolFormer * fix lint error * modify out_indices and gap * fix typo * fix lint * fix typo * fix typo in poolforemr README * fix lint * Update some paths * Refactor freeze_stages method * Add unit tests * Fix lint Co-authored-by: mzr1996 <mzr1996@163.com> * Bump version to v0.22.1 (open-mmlab#785) * [Docs] Refine API reference. (open-mmlab#774) * [Docs] Refine API reference * Add PoolFormer * [Docs] Fix docs. * [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (open-mmlab#759) * [Feature] Support VAN. (open-mmlab#739) * add van * fix config * add metafile * add test * model convert script * fix review * fix lint * fix the configs and improve docs * rm debug lines * add VAN into api Co-authored-by: Yu Zhaohui <1105212286@qq.com> * [Feature] Support DenseNet. (open-mmlab#750) * init add densenet implementation * Add config and converted models * update meta * add test for memory efficient * Add docs * add doc for jit * Update checkpoint path * Update readthedocs Co-authored-by: mzr1996 <mzr1996@163.com> * [Fix] Use symbolic link in the API reference of Chinese docs. * [Enhance] Support training on IPU and add fine-tuning configs of ViT. (open-mmlab#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai> * [Fix] Fix lint and mmcv version requirement for IPU. * Bump version to v0.23.0 (open-mmlab#809) * Refacoter Wandb hook and refine docstring Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai>

…open-mmlab#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai>

…Biases Integration) (#764) * wandb integration * visualize using wandb tables * wandb tables enhanced * Refactor MMClsWandbHook (#1) * [Enhance] Add extra dataloader settings in configs. (#752) * Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings in the `data` field to specify different arguments. * Fix bug * Fix bug * [Enhance] Improve CPE performance by reduce memory copy. (#762) * [Feature] Support resize relative position embedding in `SwinTransformer`. (#749) * [Feature]: Add resize rel pos embed * [Refactor]: Create a separated resize_rel_pos_bias_table func * [Refactor]: Refactor rel pos embed bias * [Refactor]: Move interpolate into func * Remove index buffer only when window_size changes Co-authored-by: mzr1996 <mzr1996@163.com> * [Feature] Add PoolFormer backbone and checkpoints. (#746) * add PoolFormer * fix some typos in PoolFormer * fix lint error * modify out_indices and gap * fix typo * fix lint * fix typo * fix typo in poolforemr README * fix lint * Update some paths * Refactor freeze_stages method * Add unit tests * Fix lint Co-authored-by: mzr1996 <mzr1996@163.com> * Bump version to v0.22.1 (#785) * [Docs] Refine API reference. (#774) * [Docs] Refine API reference * Add PoolFormer * [Docs] Fix docs. * [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (#759) * [Feature] Support VAN. (#739) * add van * fix config * add metafile * add test * model convert script * fix review * fix lint * fix the configs and improve docs * rm debug lines * add VAN into api Co-authored-by: Yu Zhaohui <1105212286@qq.com> * [Feature] Support DenseNet. (#750) * init add densenet implementation * Add config and converted models * update meta * add test for memory efficient * Add docs * add doc for jit * Update checkpoint path * Update readthedocs Co-authored-by: mzr1996 <mzr1996@163.com> * [Fix] Use symbolic link in the API reference of Chinese docs. * [Enhance] Support training on IPU and add fine-tuning configs of ViT. (#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai> * [Fix] Fix lint and mmcv version requirement for IPU. * Bump version to v0.23.0 (#809) * Refacoter Wandb hook and refine docstring Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai> * shuffle val data * minor updates * minor fix Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai>

…open-mmlab#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai>

…Biases Integration) (open-mmlab#764) * wandb integration * visualize using wandb tables * wandb tables enhanced * Refactor MMClsWandbHook (open-mmlab#1) * [Enhance] Add extra dataloader settings in configs. (open-mmlab#752) * Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings in the `data` field to specify different arguments. * Fix bug * Fix bug * [Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762) * [Feature] Support resize relative position embedding in `SwinTransformer`. (open-mmlab#749) * [Feature]: Add resize rel pos embed * [Refactor]: Create a separated resize_rel_pos_bias_table func * [Refactor]: Refactor rel pos embed bias * [Refactor]: Move interpolate into func * Remove index buffer only when window_size changes Co-authored-by: mzr1996 <mzr1996@163.com> * [Feature] Add PoolFormer backbone and checkpoints. (open-mmlab#746) * add PoolFormer * fix some typos in PoolFormer * fix lint error * modify out_indices and gap * fix typo * fix lint * fix typo * fix typo in poolforemr README * fix lint * Update some paths * Refactor freeze_stages method * Add unit tests * Fix lint Co-authored-by: mzr1996 <mzr1996@163.com> * Bump version to v0.22.1 (open-mmlab#785) * [Docs] Refine API reference. (open-mmlab#774) * [Docs] Refine API reference * Add PoolFormer * [Docs] Fix docs. * [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (open-mmlab#759) * [Feature] Support VAN. (open-mmlab#739) * add van * fix config * add metafile * add test * model convert script * fix review * fix lint * fix the configs and improve docs * rm debug lines * add VAN into api Co-authored-by: Yu Zhaohui <1105212286@qq.com> * [Feature] Support DenseNet. (open-mmlab#750) * init add densenet implementation * Add config and converted models * update meta * add test for memory efficient * Add docs * add doc for jit * Update checkpoint path * Update readthedocs Co-authored-by: mzr1996 <mzr1996@163.com> * [Fix] Use symbolic link in the API reference of Chinese docs. * [Enhance] Support training on IPU and add fine-tuning configs of ViT. (open-mmlab#723) * implement training and evaluation on IPU * fp16 SOTA * Tput reaches 5600 * 123 * add poptorch dataloder * change ipu_replicas to ipu-replicas * add noqa to config long line(website) * remove ipu dataloder test code * del one blank line in test_builder * refine the dataloder initialization * fix a typo * refine args for dataloder * remove an annoted line * process one more conflict * adjust code structure in mmcv.ipu * adjust ipu code structure in mmcv * IPUDataloader to IPUDataLoader * align with mmcv * adjust according to mmcv * mmcv code structre fixed Co-authored-by: hudi <dihu@graphcore.ai> * [Fix] Fix lint and mmcv version requirement for IPU. * Bump version to v0.23.0 (open-mmlab#809) * Refacoter Wandb hook and refine docstring Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai> * shuffle val data * minor updates * minor fix Co-authored-by: Ma Zerun <mzr1996@163.com> Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com> Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com> Co-authored-by: Weihao Yu <1090924009@qq.com> Co-authored-by: takuoko <to78314910@gmail.com> Co-authored-by: Yu Zhaohui <1105212286@qq.com> Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com> Co-authored-by: Hu Di <476658825@qq.com> Co-authored-by: hudi <dihu@graphcore.ai>

HuDi2018 mentioned this pull request Mar 4, 2022

implement mmcv on IPU open-mmlab/mmcv#1769

Closed

7 tasks

implement training and evaluation on IPU

d0b703b

HuDi2018 force-pushed the submit branch from 10f6a20 to d0b703b Compare March 10, 2022 01:34

fp16 SOTA

aa162f3

mzr1996 changed the base branch from master to dev March 18, 2022 02:19

HuDi2018 added 2 commits March 22, 2022 14:26

Tput reaches 5600

b10036a

123

603d556

mzr1996 reviewed Mar 23, 2022

View reviewed changes

HuDi2018 added 12 commits March 23, 2022 16:52

add poptorch dataloder

fe63a09

Merge branch 'master' into submit

4213b52

remove myconfigs in gitignore remove imagenet_bs4096_AdamW.py in config remove hdDebug.py place toHalf before collect move ipu_train_example.sh to demo

change ipu_replicas to ipu-replicas

2774d2f

add noqa to config long line(website)

f84173f

remove ipu dataloder test code

32f6129

del one blank line in test_builder

33e7d25

refine the dataloder initialization

d9ed0c7

fix a typo

259fdfd

refine args for dataloder

dbda73e

remove an annoted line

8a00327

merge to latest dev

47be422

process one more conflict

192e331

mzr1996 approved these changes Apr 1, 2022

View reviewed changes

zhouzaida mentioned this pull request Apr 7, 2022

[Feature] Support IPU open-mmlab/mmcv#1780

Closed

7 tasks

HuDi2018 added 5 commits April 7, 2022 13:21

adjust code structure in mmcv.ipu

bf8dae5

adjust ipu code structure in mmcv

25431ab

IPUDataloader to IPUDataLoader

6a3a97c

align with mmcv

9571d93

adjust according to mmcv

47f8c48

mmcv code structre fixed

be58934

mzr1996 changed the title ~~run mmclassification ViT training on IPU~~ [Enhance] Support training on IPU and add fine-tuning configs of ViT. Apr 29, 2022

mzr1996 merged commit b4eefe4 into open-mmlab:dev Apr 29, 2022

AlenLi817 mentioned this pull request May 16, 2022

The swin backbone model trained in train.py cannot be used in test.py #839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhance] Support training on IPU and add fine-tuning configs of ViT. #723

[Enhance] Support training on IPU and add fine-tuning configs of ViT. #723

HuDi2018 commented Mar 4, 2022

CLAassistant commented Mar 4, 2022 •

edited

Loading

codecov bot commented Mar 4, 2022 •

edited

Loading

Ezra-Yu commented Mar 9, 2022

mzr1996 left a comment

mzr1996 left a comment

[Enhance] Support training on IPU and add fine-tuning configs of ViT. #723

[Enhance] Support training on IPU and add fine-tuning configs of ViT. #723

Conversation

HuDi2018 commented Mar 4, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

CLAassistant commented Mar 4, 2022 • edited Loading

codecov bot commented Mar 4, 2022 • edited Loading

Codecov Report

Ezra-Yu commented Mar 9, 2022

mzr1996 left a comment

Choose a reason for hiding this comment

mzr1996 left a comment

Choose a reason for hiding this comment

CLAassistant commented Mar 4, 2022 •

edited

Loading

codecov bot commented Mar 4, 2022 •

edited

Loading