Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhance] Improve CPE performance by reduce memory copy. #762

Merged
merged 1 commit into from Apr 2, 2022
Merged

[Enhance] Improve CPE performance by reduce memory copy. #762

merged 1 commit into from Apr 2, 2022

Conversation

XiaobingSuper
Copy link
Contributor

Motivation

For position_encoding, the self.proj's input is always a non-contiguous tensor, there has a performance issue when self.stride == 1, because PyTorch always has a contiguous conversion when convolution's input is not a contiguous tensor(https://github.com/pytorch/pytorch/blob/fa09099ba35fcd42347732ca3a5f8ddaf145da1b/aten/src/ATen/native/Convolution.cpp#L1093), and add also has a similar issue(the input from convolution is a contiguous tensor, but another input is non-contiguous tensor).

Modification

This PR will do pre contiguous conversion before doing convolution and add computing.

@CLAassistant
Copy link

CLAassistant commented Apr 2, 2022 •
edited

CLA assistant check
All committers have signed the CLA.

@XiaobingSuper XiaobingSuper changed the title improve position_encoding perfromance by reduce memory copy improve position_encoding performance by reduce memory copy Apr 2, 2022
@mzr1996 mzr1996 requested a review from Ezra-Yu April 2, 2022 02:32
@codecov
Copy link

codecov bot commented Apr 2, 2022 •
edited

Codecov Report

Merging #762 (23afbca) into dev (02c8f82) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              dev     #762      +/-   ##
==========================================
+ Coverage   86.95%   86.98%   +0.03%     
==========================================
  Files         127      127              
  Lines        8071     8068       -3     
  Branches     1390     1389       -1     
==========================================
  Hits         7018     7018              
+ Misses        848      845       -3     
  Partials      205      205
Flag Coverage Δ
unittests 86.89% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmcls/models/utils/position_encoding.py 100.00% <100.00%> (ø)
mmcls/apis/train.py 19.17% <0.00%> (+0.75%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a67bc8...23afbca. Read the comment docs.

@Ezra-Yu Ezra-Yu changed the base branch from master to dev April 2, 2022 02:39
Ezra-Yu
Ezra-Yu approved these changes Apr 2, 2022
View reviewed changes
Copy link
Collaborator

@Ezra-Yu Ezra-Yu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@mzr1996 mzr1996 changed the title improve position_encoding performance by reduce memory copy [Enhance] Improve CPE performance by reduce memory copy. Apr 2, 2022
@mzr1996 mzr1996 merged commit 875195e into open-mmlab:dev Apr 2, 2022
@XiaobingSuper XiaobingSuper deleted the xiaobing/reduce_memory_copy branch April 4, 2022 11:29
ayulockin pushed a commit to ayulockin/mmclassification that referenced this pull request May 6, 2022
@mzr1996 @XiaobingSuper
@YuanLiuuuuuu @yuweihao @okotaku @Ezra-Yu @yingfhu @HuDi2018
Refactor MMClsWandbHook (#1)
8b06e18 
* [Enhance] Add extra dataloader settings in configs. (open-mmlab#752)

* Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings
in the `data` field to specify different arguments.

* Fix bug

* Fix bug

* [Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762)

* [Feature] Support resize relative position embedding in `SwinTransformer`. (open-mmlab#749)

* [Feature]: Add resize rel pos embed

* [Refactor]: Create a separated resize_rel_pos_bias_table func

* [Refactor]: Refactor rel pos embed bias

* [Refactor]: Move interpolate into func

* Remove index buffer only when window_size changes

Co-authored-by: mzr1996 <mzr1996@163.com>

* [Feature] Add PoolFormer backbone and checkpoints. (open-mmlab#746)

* add PoolFormer

* fix some typos in PoolFormer

* fix lint error

* modify out_indices and gap

* fix typo

* fix lint

* fix typo

* fix typo in poolforemr README

* fix lint

* Update some paths

* Refactor freeze_stages method

* Add unit tests

* Fix lint

Co-authored-by: mzr1996 <mzr1996@163.com>

* Bump version to v0.22.1 (open-mmlab#785)

* [Docs] Refine API reference. (open-mmlab#774)

* [Docs] Refine API reference

* Add PoolFormer

* [Docs] Fix docs.

* [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (open-mmlab#759)

* [Feature] Support VAN. (open-mmlab#739)

* add van

* fix config

* add metafile

* add test

* model convert script

* fix review

* fix lint

* fix the configs and improve docs

* rm debug lines

* add VAN into api

Co-authored-by: Yu Zhaohui <1105212286@qq.com>

* [Feature] Support DenseNet. (open-mmlab#750)

* init add densenet implementation

* Add config and converted models

* update meta

* add test for memory efficient

* Add docs

* add doc for jit

* Update checkpoint path

* Update readthedocs

Co-authored-by: mzr1996 <mzr1996@163.com>

* [Fix] Use symbolic link in the API reference of Chinese docs.

* [Enhance] Support training on IPU and add fine-tuning configs of ViT. (open-mmlab#723)

* implement training and evaluation on IPU

* fp16 SOTA

* Tput reaches 5600

* 123

* add poptorch dataloder

* change ipu_replicas to ipu-replicas

* add noqa to config long line(website)

* remove ipu dataloder test code

* del one blank line in test_builder

* refine the dataloder initialization

* fix a typo

* refine args for dataloder

* remove an annoted line

* process one more conflict

* adjust code structure in mmcv.ipu

* adjust ipu code structure in mmcv

* IPUDataloader to IPUDataLoader

* align with mmcv

* adjust according to mmcv

* mmcv code structre fixed

Co-authored-by: hudi <dihu@graphcore.ai>

* [Fix] Fix lint and mmcv version requirement for IPU.

* Bump version to v0.23.0 (open-mmlab#809)

* Refacoter Wandb hook and refine docstring

Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com>
Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com>
Co-authored-by: Weihao Yu <1090924009@qq.com>
Co-authored-by: takuoko <to78314910@gmail.com>
Co-authored-by: Yu Zhaohui <1105212286@qq.com>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
Co-authored-by: Hu Di <476658825@qq.com>
Co-authored-by: hudi <dihu@graphcore.ai>
@AlenLi817 AlenLi817 mentioned this pull request May 16, 2022
Closed
mzr1996 added a commit that referenced this pull request Jun 2, 2022
@ayulockin @mzr1996
@XiaobingSuper @YuanLiuuuuuu @yuweihao @okotaku @Ezra-Yu @yingfhu @HuDi2018
[Feature] Dedicated MMClsWandbHook for MMClassification (Weights and …
ccdbc82 
…Biases Integration) (#764)

* wandb integration

* visualize using wandb tables

* wandb tables enhanced

* Refactor MMClsWandbHook (#1)

* [Enhance] Add extra dataloader settings in configs. (#752)

* Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings
in the `data` field to specify different arguments.

* Fix bug

* Fix bug

* [Enhance] Improve CPE performance by reduce memory copy. (#762)

* [Feature] Support resize relative position embedding in `SwinTransformer`. (#749)

* [Feature]: Add resize rel pos embed

* [Refactor]: Create a separated resize_rel_pos_bias_table func

* [Refactor]: Refactor rel pos embed bias

* [Refactor]: Move interpolate into func

* Remove index buffer only when window_size changes

Co-authored-by: mzr1996 <mzr1996@163.com>

* [Feature] Add PoolFormer backbone and checkpoints. (#746)

* add PoolFormer

* fix some typos in PoolFormer

* fix lint error

* modify out_indices and gap

* fix typo

* fix lint

* fix typo

* fix typo in poolforemr README

* fix lint

* Update some paths

* Refactor freeze_stages method

* Add unit tests

* Fix lint

Co-authored-by: mzr1996 <mzr1996@163.com>

* Bump version to v0.22.1 (#785)

* [Docs] Refine API reference. (#774)

* [Docs] Refine API reference

* Add PoolFormer

* [Docs] Fix docs.

* [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (#759)

* [Feature] Support VAN. (#739)

* add van

* fix config

* add metafile

* add test

* model convert script

* fix review

* fix lint

* fix the configs and improve docs

* rm debug lines

* add VAN into api

Co-authored-by: Yu Zhaohui <1105212286@qq.com>

* [Feature] Support DenseNet. (#750)

* init add densenet implementation

* Add config and converted models

* update meta

* add test for memory efficient

* Add docs

* add doc for jit

* Update checkpoint path

* Update readthedocs

Co-authored-by: mzr1996 <mzr1996@163.com>

* [Fix] Use symbolic link in the API reference of Chinese docs.

* [Enhance] Support training on IPU and add fine-tuning configs of ViT. (#723)

* implement training and evaluation on IPU

* fp16 SOTA

* Tput reaches 5600

* 123

* add poptorch dataloder

* change ipu_replicas to ipu-replicas

* add noqa to config long line(website)

* remove ipu dataloder test code

* del one blank line in test_builder

* refine the dataloder initialization

* fix a typo

* refine args for dataloder

* remove an annoted line

* process one more conflict

* adjust code structure in mmcv.ipu

* adjust ipu code structure in mmcv

* IPUDataloader to IPUDataLoader

* align with mmcv

* adjust according to mmcv

* mmcv code structre fixed

Co-authored-by: hudi <dihu@graphcore.ai>

* [Fix] Fix lint and mmcv version requirement for IPU.

* Bump version to v0.23.0 (#809)

* Refacoter Wandb hook and refine docstring

Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com>
Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com>
Co-authored-by: Weihao Yu <1090924009@qq.com>
Co-authored-by: takuoko <to78314910@gmail.com>
Co-authored-by: Yu Zhaohui <1105212286@qq.com>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
Co-authored-by: Hu Di <476658825@qq.com>
Co-authored-by: hudi <dihu@graphcore.ai>

* shuffle val data

* minor updates

* minor fix

Co-authored-by: Ma Zerun <mzr1996@163.com>
Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com>
Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com>
Co-authored-by: Weihao Yu <1090924009@qq.com>
Co-authored-by: takuoko <to78314910@gmail.com>
Co-authored-by: Yu Zhaohui <1105212286@qq.com>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
Co-authored-by: Hu Di <476658825@qq.com>
Co-authored-by: hudi <dihu@graphcore.ai>
mzr1996 pushed a commit to mzr1996/mmpretrain that referenced this pull request Nov 24, 2022
@XiaobingSuper
[Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762
2473813 
)
mzr1996 added a commit to mzr1996/mmpretrain that referenced this pull request Nov 24, 2022
@ayulockin @mzr1996
@XiaobingSuper @YuanLiuuuuuu @yuweihao @okotaku @Ezra-Yu @yingfhu @HuDi2018
[Feature] Dedicated MMClsWandbHook for MMClassification (Weights and …
4145384 
…Biases Integration) (open-mmlab#764)

* wandb integration

* visualize using wandb tables

* wandb tables enhanced

* Refactor MMClsWandbHook (open-mmlab#1)

* [Enhance] Add extra dataloader settings in configs. (open-mmlab#752)

* Use `train_dataloader`, `val_dataloader` and `test_dataloader` settings
in the `data` field to specify different arguments.

* Fix bug

* Fix bug

* [Enhance] Improve CPE performance by reduce memory copy. (open-mmlab#762)

* [Feature] Support resize relative position embedding in `SwinTransformer`. (open-mmlab#749)

* [Feature]: Add resize rel pos embed

* [Refactor]: Create a separated resize_rel_pos_bias_table func

* [Refactor]: Refactor rel pos embed bias

* [Refactor]: Move interpolate into func

* Remove index buffer only when window_size changes

Co-authored-by: mzr1996 <mzr1996@163.com>

* [Feature] Add PoolFormer backbone and checkpoints. (open-mmlab#746)

* add PoolFormer

* fix some typos in PoolFormer

* fix lint error

* modify out_indices and gap

* fix typo

* fix lint

* fix typo

* fix typo in poolforemr README

* fix lint

* Update some paths

* Refactor freeze_stages method

* Add unit tests

* Fix lint

Co-authored-by: mzr1996 <mzr1996@163.com>

* Bump version to v0.22.1 (open-mmlab#785)

* [Docs] Refine API reference. (open-mmlab#774)

* [Docs] Refine API reference

* Add PoolFormer

* [Docs] Fix docs.

* [Enhance] Reduce the memory usage of unit tests for Swin-Transformer. (open-mmlab#759)

* [Feature] Support VAN. (open-mmlab#739)

* add van

* fix config

* add metafile

* add test

* model convert script

* fix review

* fix lint

* fix the configs and improve docs

* rm debug lines

* add VAN into api

Co-authored-by: Yu Zhaohui <1105212286@qq.com>

* [Feature] Support DenseNet. (open-mmlab#750)

* init add densenet implementation

* Add config and converted models

* update meta

* add test for memory efficient

* Add docs

* add doc for jit

* Update checkpoint path

* Update readthedocs

Co-authored-by: mzr1996 <mzr1996@163.com>

* [Fix] Use symbolic link in the API reference of Chinese docs.

* [Enhance] Support training on IPU and add fine-tuning configs of ViT. (open-mmlab#723)

* implement training and evaluation on IPU

* fp16 SOTA

* Tput reaches 5600

* 123

* add poptorch dataloder

* change ipu_replicas to ipu-replicas

* add noqa to config long line(website)

* remove ipu dataloder test code

* del one blank line in test_builder

* refine the dataloder initialization

* fix a typo

* refine args for dataloder

* remove an annoted line

* process one more conflict

* adjust code structure in mmcv.ipu

* adjust ipu code structure in mmcv

* IPUDataloader to IPUDataLoader

* align with mmcv

* adjust according to mmcv

* mmcv code structre fixed

Co-authored-by: hudi <dihu@graphcore.ai>

* [Fix] Fix lint and mmcv version requirement for IPU.

* Bump version to v0.23.0 (open-mmlab#809)

* Refacoter Wandb hook and refine docstring

Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com>
Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com>
Co-authored-by: Weihao Yu <1090924009@qq.com>
Co-authored-by: takuoko <to78314910@gmail.com>
Co-authored-by: Yu Zhaohui <1105212286@qq.com>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
Co-authored-by: Hu Di <476658825@qq.com>
Co-authored-by: hudi <dihu@graphcore.ai>

* shuffle val data

* minor updates

* minor fix

Co-authored-by: Ma Zerun <mzr1996@163.com>
Co-authored-by: XiaobingZhang <xiaobing.zhang@intel.com>
Co-authored-by: Yuan Liu <30762564+YuanLiuuuuuu@users.noreply.github.com>
Co-authored-by: Weihao Yu <1090924009@qq.com>
Co-authored-by: takuoko <to78314910@gmail.com>
Co-authored-by: Yu Zhaohui <1105212286@qq.com>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
Co-authored-by: Hu Di <476658825@qq.com>
Co-authored-by: hudi <dihu@graphcore.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ezra-Yu Ezra-Yu Ezra-Yu approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@XiaobingSuper @CLAassistant @Ezra-Yu @mzr1996