Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] Support MultiScaleDeformableAttention with AMP #2541

Merged
merged 6 commits into from
Feb 17, 2023
Merged

[Enhancement] Support MultiScaleDeformableAttention with AMP #2541

merged 6 commits into from
Feb 17, 2023

Conversation

nijkah
Copy link
Contributor

@nijkah nijkah commented Jan 10, 2023 •
edited
Loading

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Currently, MultiScaleDeformableAttention with AMP is not supported.

RuntimeError: "ms_deform_attn_forward_cuda" not implemented for 'Half'

Modification

Following https://discuss.pytorch.org/t/how-can-i-write-the-cuda-code-to-support-fp16-calculation/107181 and DeformConv2d,
apply AT_DISPATCH_FLOATING_TYPES_AND_HALF and solve the type mismatch.

Checklist

Before PR:

  • I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
  • Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
  • Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
  • New functionalities are covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with some of those projects, like MMDet or MMCls.
  • CLA has been signed and all committers have signed the CLA in this PR.

@nijkah nijkah mentioned this pull request Jan 10, 2023
Merged
@nijkah
Copy link
Contributor Author

nijkah commented Jan 11, 2023

Should I post a new PR for mmcv 2.x versino?

@zhouzaida
Copy link
Collaborator

Should I post a new PR for mmcv 2.x versino?

Thanks for your contribution. I will cherry-pick this commit to 2.x.

@zhouzaida zhouzaida changed the title [Enhancement] Support FP16 for MultiScaleDeformableAttention [Enhancement] Support MultiScaleDeformableAttention with AMP Jan 18, 2023
@zhouzaida
Copy link
Collaborator

Please add unit tests to cover this modification. You can refer to https://github.com/open-mmlab/mmcv/blob/master/tests/test_ops/test_deform_conv.py.

@zhouzaida zhouzaida self-requested a review January 18, 2023 07:10
@nijkah
Add UT
8ed8896 
Author:    nijkah <nijkah@gmail.com>
@nijkah
Copy link
Contributor Author

nijkah commented Jan 30, 2023

@zhouzaida Sorry for the late response. I added UT.
But, I'm not sure about the precision threshold. Can I ask you to check it?

@zhouzaida
Copy link
Collaborator

@zhouzaida Sorry for the late response. I added UT. But, I'm not sure about the precision threshold. Can I ask you to check it?

@nijkah Sorry for my late reply. The precision threshold is OK.

@zhouzaida zhouzaida merged commit 8e8ab22 into open-mmlab:master Feb 17, 2023
@nijkah nijkah deleted the feature/ms_deform_half branch February 17, 2023 13:18
zhouzaida added a commit to zhouzaida/mmcv that referenced this pull request Mar 20, 2023
@nijkah @zhouzaida
[Enhancement] Support MultiScaleDeformableAttention with AMP (open-mm…
9dd51b8 
…lab#2541)

* [Enhance] Support FP16 for MSDeformAttn

* [Fix] Data type mismatch

* Update mmcv/ops/multi_scale_deform_attn.py

* Add UT

Author:    nijkah <nijkah@gmail.com>

* Add cuda available condition

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
zhouzaida added a commit to zhouzaida/mmcv that referenced this pull request Mar 20, 2023
@nijkah @zhouzaida
[Enhancement] Support MultiScaleDeformableAttention with AMP (open-mm…
964afd2 
…lab#2541)

* [Enhance] Support FP16 for MSDeformAttn

* [Fix] Data type mismatch

* Update mmcv/ops/multi_scale_deform_attn.py

* Add UT

Author:    nijkah <nijkah@gmail.com>

* Add cuda available condition

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
zhouzaida added a commit that referenced this pull request Mar 20, 2023
@nijkah @zhouzaida
[Enhancement] Support MultiScaleDeformableAttention with AMP (#2541)
4ad7959 
* [Enhance] Support FP16 for MSDeformAttn

* [Fix] Data type mismatch

* Update mmcv/ops/multi_scale_deform_attn.py

* Add UT

Author:    nijkah <nijkah@gmail.com>

* Add cuda available condition

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
CokeDong pushed a commit to CokeDong/mmcv that referenced this pull request Apr 24, 2023
@nijkah @zhouzaida @CokeDong
[Enhancement] Support MultiScaleDeformableAttention with AMP (open-mm…
1a05f56 
…lab#2541)

* [Enhance] Support FP16 for MSDeformAttn

* [Fix] Data type mismatch

* Update mmcv/ops/multi_scale_deform_attn.py

* Add UT

Author:    nijkah <nijkah@gmail.com>

* Add cuda available condition

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
tyomj pushed a commit to tyomj/mmcv that referenced this pull request May 8, 2023
@nijkah @zhouzaida @akozlov-outrider
[Enhancement] Support MultiScaleDeformableAttention with AMP (open-mm…
2f34f71 
…lab#2541)

* [Enhance] Support FP16 for MSDeformAttn

* [Fix] Data type mismatch

* Update mmcv/ops/multi_scale_deform_attn.py

* Add UT

Author:    nijkah <nijkah@gmail.com>

* Add cuda available condition

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
@Dawn-bin Dawn-bin mentioned this pull request Aug 1, 2023
Open
fwenguang pushed a commit to fwenguang/mmcv that referenced this pull request Dec 19, 2023
@nijkah @zhouzaida @ClowDragon
[Enhancement] Support MultiScaleDeformableAttention with AMP (open-mm…
27c4b97 
…lab#2541)

* [Enhance] Support FP16 for MSDeformAttn

* [Fix] Data type mismatch

* Update mmcv/ops/multi_scale_deform_attn.py

* Add UT

Author:    nijkah <nijkah@gmail.com>

* Add cuda available condition

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhouzaida zhouzaida zhouzaida approved these changes

Assignees

@ice-tong ice-tong

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@nijkah @zhouzaida @ice-tong