Added pow() on CPU for float16 & bfloat16 #50999

imaginary-person · 2021-01-24T00:18:01Z

Added the functionality desired in #50789.

Summary

Added support for pow() on CPU for float16 (Half) and bfloat16 types.
Both pow(Tensor, Scalar) and pow(Tensor, Tensor) are now supported for the aforementioned types.
However autograd isn't supported for Float16 on CPU yet, as log_vml_cpu can't be enabled for it.
@heitorschueroff added pow_tensor_scalar_optimized_kernel to refactor & simplify PowKernel.cpp.
It provides a common path for all the complex types & floating point types (except Float16, due to lack of complete AVX2 vectorization support for it). It replaced code that had previously been duplicated for (float, double) and complex types,
so PowKernel.cpp looks a lot cleaner now.
Enabled (unskipped) some tests for erf, erfc,erfinv, linalg.norm and linalg.vector.norm which were being skipped earlier due to pow() not having been implemented for float16 & bfloat16.
Added an OpInfo for pow() & enabled some test cases for pow().
Extended the coverage of existing tests for pow in test_binary_ufuncs.py in order to enable comparison with numpy, even with discontiguous tensors, and added a test to ensure that a runtime error is raised for pow's inplace variant if resizing the base tensor is required during its invocation.
Added float16 & bfloat16 to square's dtype lists in its UnaryUfuncInfo.

facebook-github-bot · 2021-01-24T00:18:13Z

💊 CI failures summary and remediations

As of commit 3457e35 (more details on the Dr. CI page):

4/4 failures introduced in this PR

4 failures not recognized by patterns:

Job	Step	Action
^clang-tidy	^Unknown	🔁 rerun
^flake8-py3	^Unknown	🔁 rerun
^mypy	^Unknown	🔁 rerun
^quick-checks	^Unknown	🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

aten/src/ATen/native/cpu/PowKernel.cpp

mruberry · 2021-01-25T17:37:57Z

@imaginary-person do me a favor and ping me tomorrow or Wednesday on this PR and I'll take a look.

codecov · 2021-01-25T22:04:33Z

Codecov Report

Merging #50999 (227e771) into master (9e6877c) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 227e771 differs from pull request most recent head 7274779. Consider uploading reports for the commit 7274779 to get more accurate results

@@           Coverage Diff           @@
##           master   #50999   +/-   ##
=======================================
  Coverage   77.45%   77.46%           
=======================================
  Files        1894     1894           
  Lines      186403   186437   +34     
=======================================
+ Hits       144374   144416   +42     
+ Misses      42029    42021    -8

c10/util/BFloat16-inl.h

torch/testing/_internal/common_methods_invocations.py

aten/src/ATen/native/cpu/PowKernel.cpp

test/test_binary_ufuncs.py

test/test_torch.py

torch/testing/_internal/common_methods_invocations.py

mruberry

Hey @imaginary-person!

There's a lot of good stuff going on here. An OpInfo needs to be added for pow, and the legacy pow sample inputs removed from method_tests in common_methods_invocations.py. Implementing an OpInfo for pow will also let you remove those legacy test_torch.py pow sample inputs and not have to worry about updating them.

As for the pow function itself, can we really not simplify the implementation to avoid tripling its size? pow has been an extremely tricky function to get right, and I'm worried about making it even harder to maintain.

heitorschueroff

LGTM! @anjali411 do you want to check the complex part before I land this?

Thanks for the great work on this PR @imaginary-person

imaginary-person · 2021-03-31T19:30:18Z

@heitorschueroff, thanks a lot for your & @mruberry's enormous help & patience with this PR!
BTW, I haven't unskipped test_int_and_float_pow for other dtypes (except int8) on ROCm yet.

[skip ci]

facebook-github-bot · 2021-03-31T20:17:52Z

@heitorschueroff has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

aten/src/ATen/native/cpu/PowKernel.cpp

Removed scalar_t = decltype(c10::impl::ScalarTypeToCPPType<ScalarType::Half>::t), as the AT_DISPATCH_FLOATING_TYPES_AND macro does this assignment anyway.

facebook-github-bot · 2021-04-01T03:37:27Z

@heitorschueroff has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ngimel

lgtm

facebook-github-bot · 2021-04-02T22:58:35Z

@heitorschueroff merged this pull request in 6d030c1.

heitorschueroff · 2021-04-02T23:16:44Z

@imaginary-person This was not a trivial task and the refactor + tests you added are a great contribution to the project. Thank you!

peterjc123 · 2021-04-03T15:31:08Z

This PR is causing the majority of the jobs to fail in CI, as can be observed in https://ezyang.github.io/pytorch-ci-hud/build2/pytorch-master.
Example test runs:
https://app.circleci.com/pipelines/github/pytorch/pytorch/295605/workflows/6f0cdf22-772d-4bf4-9ee2-9ad41ed2a668/jobs/12095451
https://app.circleci.com/pipelines/github/pytorch/pytorch/295605/workflows/6f0cdf22-772d-4bf4-9ee2-9ad41ed2a668/jobs/12096482
https://app.circleci.com/pipelines/github/pytorch/pytorch/295605/workflows/6f0cdf22-772d-4bf4-9ee2-9ad41ed2a668/jobs/12096722

facebook-github-bot · 2021-04-03T18:07:45Z

This pull request has been reverted by 8377e62.

imaginary-person · 2021-04-03T18:13:28Z

@malfet @peterjc123,

Sorry for the inconvenience!
#54949 has affected the failures. I haven't checked why yet.
I'll check & re-submit this PR for relanding after making any required changes.

EDIT: #54949 caught a bug in this PR. For 4 of the 9 bool sample inputs, the dtype wasn't actually bool! 😞
So, this line had to be modified, with dtype being changed to dtype=torch.bool:

https://github.com/imaginary-person/pytorch-1/blob/e451285877225d3079faf5a8001e52ca22dc64ab/torch/testing/_internal/common_methods_invocations.py#L1607

Get changes from main repo

Summary: #### Reason for relanding Line 1607 of `torch/testing/_internal/common_methods_invocations.py` of #50999 had `dtype` instead of `dtype=torch.bool`, so 4 of the 9 sample inputs for `bool` had incorrect dtype. This bug was caught by #54949. 1. Added support for pow() on CPU for `float16` (`Half`) and `bfloat16` types. Both `pow(Tensor, Scalar)` and `pow(Tensor, Tensor)` are now supported for the aforementioned types. However autograd isn't supported for `Float16` on CPU yet, as `log_vml_cpu` can't be enabled for it. 2. heitorschueroff added `pow_tensor_scalar_optimized_kernel` to refactor & simplify `PowKernel.cpp`. It provides a common path for all the complex types & floating point types (except Float16, due to lack of complete AVX2 vectorization support for it). It replaced code that had previously been duplicated for (float, double) and complex types, so PowKernel.cpp looks a lot cleaner now. 3. Enabled (unskipped) some tests for `erf`, `erfc`,`erfinv`, `tan` and `linalg.vector.norm` which were being skipped earlier due to `pow()` not having been implemented for `float16` & `bfloat16`. 4. Added an OpInfo for `pow()` & enabled some test cases for `pow()`. 5. Extended the coverage of existing tests for `pow` in `test_binary_ufuncs.py` in order to enable comparison with `numpy`, even with discontiguous tensors, and added a test to ensure that a runtime error is raised for `pow`'s inplace variant if resizing the base tensor is required during its invocation. 6. Added `float16` & `bfloat16` to `square`'s dtype lists in its `UnaryUfuncInfo`. 7. Removed redundant `dtypesIfCPU` and `dtypesIfCUDA` from `OpInfo`s where they are equal to `dtypes`. Pull Request resolved: #55280 Reviewed By: jbschlosser Differential Revision: D27591772 Pulled By: heitorschueroff fbshipit-source-id: c7420811b32595bb3353149a61e54a73f2eb352b

…orch#55280) Summary: #### Reason for relanding Line 1607 of `torch/testing/_internal/common_methods_invocations.py` of pytorch#50999 had `dtype` instead of `dtype=torch.bool`, so 4 of the 9 sample inputs for `bool` had incorrect dtype. This bug was caught by pytorch#54949. 1. Added support for pow() on CPU for `float16` (`Half`) and `bfloat16` types. Both `pow(Tensor, Scalar)` and `pow(Tensor, Tensor)` are now supported for the aforementioned types. However autograd isn't supported for `Float16` on CPU yet, as `log_vml_cpu` can't be enabled for it. 2. heitorschueroff added `pow_tensor_scalar_optimized_kernel` to refactor & simplify `PowKernel.cpp`. It provides a common path for all the complex types & floating point types (except Float16, due to lack of complete AVX2 vectorization support for it). It replaced code that had previously been duplicated for (float, double) and complex types, so PowKernel.cpp looks a lot cleaner now. 3. Enabled (unskipped) some tests for `erf`, `erfc`,`erfinv`, `tan` and `linalg.vector.norm` which were being skipped earlier due to `pow()` not having been implemented for `float16` & `bfloat16`. 4. Added an OpInfo for `pow()` & enabled some test cases for `pow()`. 5. Extended the coverage of existing tests for `pow` in `test_binary_ufuncs.py` in order to enable comparison with `numpy`, even with discontiguous tensors, and added a test to ensure that a runtime error is raised for `pow`'s inplace variant if resizing the base tensor is required during its invocation. 6. Added `float16` & `bfloat16` to `square`'s dtype lists in its `UnaryUfuncInfo`. 7. Removed redundant `dtypesIfCPU` and `dtypesIfCUDA` from `OpInfo`s where they are equal to `dtypes`. Pull Request resolved: pytorch#55280 Reviewed By: jbschlosser Differential Revision: D27591772 Pulled By: heitorschueroff fbshipit-source-id: c7420811b32595bb3353149a61e54a73f2eb352b

facebook-github-bot added the cla signed label Jan 24, 2021

This comment has been minimized.

Sign in to view

imaginary-person commented Jan 24, 2021

View reviewed changes

aten/src/ATen/native/cpu/PowKernel.cpp Outdated Show resolved Hide resolved

pytorchbot added the open source label Jan 24, 2021

This comment has been minimized.

Sign in to view

imaginary-person marked this pull request as ready for review January 25, 2021 17:29

imaginary-person mentioned this pull request Jan 25, 2021

test_variant_consistency_jit for BFloat16 are skipped in a confusing way #48978

Closed

mruberry self-requested a review January 25, 2021 17:36

mruberry added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jan 25, 2021

This comment has been minimized.

Sign in to view

imaginary-person marked this pull request as draft January 25, 2021 17:43

This comment has been minimized.

Sign in to view

imaginary-person commented Jan 25, 2021

View reviewed changes

c10/util/BFloat16-inl.h Outdated Show resolved Hide resolved

imaginary-person commented Jan 25, 2021

View reviewed changes

torch/testing/_internal/common_methods_invocations.py Outdated Show resolved Hide resolved

imaginary-person marked this pull request as ready for review January 26, 2021 04:52

This comment has been minimized.

Sign in to view

imaginary-person mentioned this pull request Jan 28, 2021

Add missing -inf order for linalg.norm OpInfo #51233

Closed

imaginary-person marked this pull request as draft January 28, 2021 18:49

imaginary-person force-pushed the imaginary-person-pow-float16-bfloat16 branch from 96e2d44 to 5c57e01 Compare January 30, 2021 00:52

imaginary-person marked this pull request as ready for review January 30, 2021 03:59

This comment has been minimized.

Sign in to view

mruberry reviewed Feb 1, 2021

View reviewed changes

aten/src/ATen/native/cpu/PowKernel.cpp Outdated Show resolved Hide resolved

mruberry reviewed Feb 1, 2021

View reviewed changes

test/test_binary_ufuncs.py Outdated Show resolved Hide resolved

mruberry reviewed Feb 1, 2021

View reviewed changes

test/test_torch.py Outdated Show resolved Hide resolved

mruberry reviewed Feb 1, 2021

View reviewed changes

torch/testing/_internal/common_methods_invocations.py Outdated Show resolved Hide resolved

mruberry reviewed Feb 1, 2021

View reviewed changes

heitorschueroff approved these changes Mar 31, 2021

View reviewed changes

anjali411 approved these changes Mar 31, 2021

View reviewed changes

[skip ci] Updated comment in PowKernel.cpp

23d14b8

[skip ci]

heitorschueroff reviewed Apr 1, 2021

View reviewed changes

aten/src/ATen/native/cpu/PowKernel.cpp Outdated Show resolved Hide resolved

Remove redundant scalar_t assignment for BFloat16

e451285

Removed scalar_t = decltype(c10::impl::ScalarTypeToCPPType<ScalarType::Half>::t), as the AT_DISPATCH_FLOATING_TYPES_AND macro does this assignment anyway.

heitorschueroff requested a review from ngimel April 1, 2021 17:47

ngimel approved these changes Apr 2, 2021

View reviewed changes

facebook-github-bot closed this in 6d030c1 Apr 2, 2021

facebook-github-bot added the Merged label Apr 2, 2021

facebook-github-bot added the Reverted label Apr 3, 2021

Get changes from main repo

3457e35

Get changes from main repo

imaginary-person mentioned this pull request Apr 3, 2021

Reland #50999 (Added pow() on CPU for float16 & bfloat16) #55280

Closed

This comment has been minimized.

Sign in to view

heitorschueroff reopened this Apr 5, 2021

heitorschueroff closed this Apr 5, 2021

anjali411 mentioned this pull request Apr 5, 2021

Adopt native complex dtype in griffnlim pytorch/audio#1368

Merged

3 tasks

WhoAteDaCake mentioned this pull request Oct 27, 2021

[LayoutReader] RuntimeError: "pow" not implemented for 'Half' microsoft/unilm#489

Closed

2 tasks

Added pow() on CPU for float16 & bfloat16 #50999

Added pow() on CPU for float16 & bfloat16 #50999

Uh oh!

Conversation

imaginary-person commented Jan 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

facebook-github-bot commented Jan 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

4 failures not recognized by patterns:

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

This comment has been minimized.

mruberry commented Jan 25, 2021

Uh oh!

This comment has been minimized.

codecov bot commented Jan 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mruberry left a comment

Choose a reason for hiding this comment

Uh oh!

heitorschueroff left a comment

Choose a reason for hiding this comment

Uh oh!

imaginary-person commented Mar 31, 2021

Uh oh!

facebook-github-bot commented Mar 31, 2021

Uh oh!

Uh oh!

facebook-github-bot commented Apr 1, 2021

Uh oh!

ngimel left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Apr 2, 2021

Uh oh!

heitorschueroff commented Apr 2, 2021

Uh oh!

peterjc123 commented Apr 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Apr 3, 2021

Uh oh!

imaginary-person commented Apr 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

imaginary-person commented Jan 24, 2021 •

edited

Loading

facebook-github-bot commented Jan 24, 2021 •

edited

Loading

codecov bot commented Jan 25, 2021 •

edited

Loading

peterjc123 commented Apr 3, 2021 •

edited

Loading

imaginary-person commented Apr 3, 2021 •

edited

Loading