Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support kl_div function with bfloat16 #77375

Closed
rosrad opened this issue May 12, 2022 · 5 comments
Closed

support kl_div function with bfloat16 #77375

rosrad opened this issue May 12, 2022 · 5 comments
Labels
enhancement Not as big of a feature, but technically not a bug. Should be easy to fix module: bfloat16 module: nn Related to torch.nn triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@rosrad
Copy link

rosrad commented May 12, 2022 •
edited by pytorch-bot bot
Loading

馃殌 The feature, motivation and pitch

We are trying to switch our model training from the fp16 to bfloat16 precision. This change can be good for the CE loss functions, however it failed on the kl_div function due to non implementation.

here is the error log:
"kl_div_backward_cuda" not implemented for 'BFloat16'

Alternatives

No response

Additional context

No response

cc @albanD @mruberry @jbschlosser @walterddr @kshitij12345
@samdow samdow added module: nn Related to torch.nn triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module module: bfloat16 enhancement Not as big of a feature, but technically not a bug. Should be easy to fix labels May 13, 2022
@ngimel
Copy link
Collaborator

ngimel commented May 14, 2022

cc @ptrblck, can someone send a PR implementing this please?

@ptrblck
Copy link
Collaborator

ptrblck commented May 16, 2022

@Aidyn-A could you take a look at this, please?

@Aidyn-A
Copy link
Collaborator

Aidyn-A commented May 16, 2022

Sure, I will take care of this.

@Aidyn-A Aidyn-A mentioned this issue May 17, 2022
Closed
pytorchmergebot pushed a commit that referenced this issue May 24, 2022
@Aidyn-A @pytorchmergebot
add bfloat16 support for kl_div_backward_cuda (#77676)
f37ce94 
This PR adds a feature requested in issue #77375.
`kl_div_backward_cuda` now supports `bfloat16`

cc @ngimel @ptrblck @rosrad

Pull Request resolved: #77676
Approved by: https://github.com/jbschlosser
facebook-github-bot pushed a commit that referenced this issue May 26, 2022
@Aidyn-A @facebook-github-bot
add bfloat16 support for kl_div_backward_cuda (#77676) (#77676)
7237dbb 
Summary:
This PR adds a feature requested in issue #77375.
`kl_div_backward_cuda` now supports `bfloat16`

cc ngimel ptrblck rosrad

Pull Request resolved: #77676
Approved by: https://github.com/jbschlosser

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/f37ce948ff25bf70a52ab4b327c82925ddb4aa86

Reviewed By: mehtanirav

Differential Revision: D36668740

fbshipit-source-id: 0f171ac2fdb66931f0a7ffe73a97517fe2abad02
@boren-ms
Copy link

Thanks a lot for the timely support.

@jbschlosser
Copy link
Contributor

Closing as addressed in #77676.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Not as big of a feature, but technically not a bug. Should be easy to fix module: bfloat16 module: nn Related to torch.nn triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@rosrad @ptrblck @ngimel @samdow @boren-ms @Aidyn-A @jbschlosser