CUDA BFloat16 support of clamp, remainder, lshift, rshift #45247

zasdfgbnm · 2020-09-24T00:16:25Z

Add CUDA BFloat16 support of clamp, remainder, lshift, rshift

dr-ci · 2020-09-24T00:57:10Z

💊 CI failures summary and remediations

As of commit de95b58 (more details on the Dr. CI page):

1/2 failures introduced in this PR
1/2 broken upstream at merge base dc9e9c1 on Sep 25 from 11:03am to 2:46pm PDT (11 commits; d1a1161 - 04be420)

XLA failure

Job pytorch_xla_linux_bionic_py3_6_clang9_test is failing. Please create an issue with title prefixed by [PT_BREAK] in pytorch/xla and link to to this PR. If you have questions, please reach out to @ailzhang / @dlibenzi / @JackCaoG.

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is newer than viable/strict, you can try basing on an older, stable commit:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase --onto FETCH_HEAD $(git merge-base origin/master HEAD)

If your commit is older than viable/strict:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

Check out the recency history of this "viable master" tracking branch.

pytorch_linux_backward_compatibility_check_test on Sep 25 from 11:03am to 2:46pm PDT (11 commits; d1a1161 - 04be420)
- 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 16 times.

dr-ci · 2020-09-24T00:58:26Z

💊 CI failures summary and remediations

As of commit 9ae9ad1 (more details on the Dr. CI page):

Commit 9ae9ad1 was recently pushed. Waiting for builds...

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 2 times.

izdeby · 2020-09-25T15:20:11Z

test/test_torch.py

-    ('clamp_min', '', _medium_2d, lambda t, d: [1], 1e-2, 1e-2, 1e-5, _types, [torch.bfloat16]),
-    ('clamp_max', '', _medium_2d, lambda t, d: [1], 1e-2, 1e-2, 1e-5, _types, [torch.bfloat16]),
+    ('clamp_min', '', _medium_2d, lambda t, d: [1], 1e-2, 1e-2, 1e-5,
+        torch.testing.get_all_dtypes(include_complex=False, include_bool=False), [torch.bfloat16]),


you can pass include_bfloat16=True here

izdeby · 2020-09-25T15:20:57Z

aten/src/ATen/native/cuda/BinaryShiftOpsKernels.cu

-    AT_DISPATCH_FLOATING_TYPES_AND(ScalarType::Half, iter.dtype(), "lshift_cuda", [&]() {
+      iter.dtype() == ScalarType::Half ||
+      iter.dtype() == ScalarType::BFloat16) {
+    AT_DISPATCH_FLOATING_TYPES_AND2(ScalarType::Half, ScalarType::BFloat16, iter.dtype(), "lshift_cuda", [&]() {


how is this being tested?

It is tested by

('__lshift__', '', lambda t, d: torch.pow(2, torch.arange(1, 5).to(dtype=_convert_t(t, d), device=d)), lambda t, d: [2], 1e-3, 1e-5, 1e-3, _signed_types, _cpu_types, False), ('__rshift__', '', lambda t, d: torch.pow(2, torch.arange(3, 7).to(dtype=_convert_t(t, d), device=d)), lambda t, d: [2], 1e-3, 1e-5, 1e-3, _signed_types, _cpu_types, False),

where the _signed_types is modified in https://github.com/pytorch/pytorch/pull/45247/files#diff-9996665f82f52030836eb8657057cfadR19601-R19604 to add bfloat16

izdeby · 2020-09-25T15:23:04Z

test/test_torch.py

+    ('remainder', 'value', _small_3d, lambda t, d: [3], 1e-1, 1e-2, 1e-5, _signed_types),
+    ('remainder', 'negative_value', _small_3d, lambda t, d: [-3], 1e-1, 1e-2, 1e-5, _signed_types),
    ('remainder', 'tensor', _small_3d,
        lambda t, d: [_small_3d(t, d, has_zeros=False)],
-        1e-1, 1e-5, 1e-5, _signed_types),
+        1e-1, 1e-2, 1e-5, _signed_types),
    ('remainder', 'negative_tensor', _small_3d,
        lambda t, d: [0 - _small_3d(t, d, has_zeros=False)],
-        1e-1, 1e-5, 1e-5, _signed_types),
+        1e-1, 1e-2, 1e-5, _signed_types),


None of these is testing BFloat16 because

_signed_types = [ torch.half, torch.float, torch.double, torch.int8, torch.short, torch.int, torch.long ]

The change is here: https://github.com/pytorch/pytorch/pull/45247/files#diff-9996665f82f52030836eb8657057cfadR19601-R19604
I added bfloat16 to _signed_types.

izdeby

Please add bfloat16 tests.
Update description to something more meaningful.
Update title to mention Bfloat16

…ntwise2

… bfloat-elementwise2

ngimel · 2020-10-04T05:46:11Z

@izdeby's comments are addressed, but xla failures are real - @JackCaoG do you guys want to disable those tests or fix?

…ntwise2

codecov · 2020-10-07T21:46:01Z

Codecov Report

Merging #45247 into master will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #45247   +/-   ##
=======================================
  Coverage   68.25%   68.25%           
=======================================
  Files         410      410           
  Lines       53232    53232           
=======================================
+ Hits        36335    36336    +1     
+ Misses      16897    16896    -1

Impacted Files	Coverage Δ
torch/testing/_internal/expecttest.py	`78.57% <0.00%> (+1.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83d2c9a...9ae9ad1. Read the comment docs.

facebook-github-bot

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-10-08T04:11:25Z

@ngimel merged this pull request in 903acc6.

zasdfgbnm added 2 commits September 23, 2020 17:16

CUDA BFloat support of clamp, remainder, lshift, rshift

1d0e860

flake8

3442ce3

pytorchbot added the open source label Sep 24, 2020

Merge branch 'master' into bfloat-elementwise2

6493d82

zou3519 requested review from ngimel and izdeby September 25, 2020 14:32

zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Sep 25, 2020

izdeby reviewed Sep 25, 2020

View reviewed changes

izdeby suggested changes Sep 25, 2020

View reviewed changes

zasdfgbnm added 3 commits September 25, 2020 11:29

Merge branch 'master' of github.com:pytorch/pytorch into bfloat-eleme…

1683e1e

…ntwise2

include_bfloat16=True

c2c9d50

Merge branch 'bfloat-elementwise2' of github.com:pytorch/pytorch into…

de95b58

… bfloat-elementwise2

ngimel changed the title ~~CUDA BFloat support of clamp, remainder, lshift, rshift~~ CUDA BFloat16 support of clamp, remainder, lshift, rshift Oct 4, 2020

Merge branch 'master' of github.com:pytorch/pytorch into bfloat-eleme…

740c393

…ntwise2

zasdfgbnm mentioned this pull request Oct 6, 2020

[PT_BREAK] value cannot be converted to type at::BFloat16 without overflow: -3.40282e+38 pytorch/xla#2535

Closed

Merge branch 'master' of github.com:pytorch/pytorch into bfloat-eleme…

9ae9ad1

…ntwise2

facebook-github-bot reviewed Oct 7, 2020

View reviewed changes

ngimel approved these changes Oct 7, 2020

View reviewed changes

facebook-github-bot closed this in 903acc6 Oct 8, 2020

facebook-github-bot added the Merged label Oct 8, 2020

zasdfgbnm deleted the bfloat-elementwise2 branch October 8, 2020 04:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA BFloat16 support of clamp, remainder, lshift, rshift #45247

CUDA BFloat16 support of clamp, remainder, lshift, rshift #45247

zasdfgbnm commented Sep 24, 2020 •

edited

dr-ci bot commented Sep 24, 2020 •

edited

dr-ci bot commented Sep 24, 2020 •

edited

izdeby Sep 25, 2020

zasdfgbnm Sep 25, 2020

izdeby Sep 25, 2020

zasdfgbnm Sep 25, 2020 •

edited

izdeby Sep 25, 2020

zasdfgbnm Sep 25, 2020 •

edited

izdeby left a comment •

edited

ngimel commented Oct 4, 2020

codecov bot commented Oct 7, 2020 •

edited

facebook-github-bot left a comment

facebook-github-bot commented Oct 8, 2020

CUDA BFloat16 support of clamp, remainder, lshift, rshift #45247

CUDA BFloat16 support of clamp, remainder, lshift, rshift #45247

Conversation

zasdfgbnm commented Sep 24, 2020 • edited

dr-ci bot commented Sep 24, 2020 • edited

💊 CI failures summary and remediations

XLA failure

🚧 1 fixed upstream failure:

dr-ci bot commented Sep 24, 2020 • edited

💊 CI failures summary and remediations

izdeby Sep 25, 2020

Choose a reason for hiding this comment

zasdfgbnm Sep 25, 2020

Choose a reason for hiding this comment

izdeby Sep 25, 2020

Choose a reason for hiding this comment

zasdfgbnm Sep 25, 2020 • edited

Choose a reason for hiding this comment

izdeby Sep 25, 2020

Choose a reason for hiding this comment

zasdfgbnm Sep 25, 2020 • edited

Choose a reason for hiding this comment

izdeby left a comment • edited

Choose a reason for hiding this comment

ngimel commented Oct 4, 2020

codecov bot commented Oct 7, 2020 • edited

Codecov Report

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Oct 8, 2020

zasdfgbnm commented Sep 24, 2020 •

edited

dr-ci bot commented Sep 24, 2020 •

edited

dr-ci bot commented Sep 24, 2020 •

edited

zasdfgbnm Sep 25, 2020 •

edited

zasdfgbnm Sep 25, 2020 •

edited

izdeby left a comment •

edited

codecov bot commented Oct 7, 2020 •

edited