Enable Add, sub, mul, and div on CPU for bfloat16 type. #22851

izdeby · 2019-07-14T21:19:50Z

Enable Add, sub, mul, and div on CPU for bfloat16 type.
Tested via unit tests.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

aten/src/ATen/cpu/vec256/vec256_base.h

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

aten/src/ATen/native/cpu/BinaryOpsKernel.cpp

c10/core/ScalarType.h

test/test_torch.py

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

aten/src/ATen/native/cpu/BinaryOpsKernel.cpp

test/test_torch.py

gchanan

a few things plus I can't reproduce the "= {0};" change.

c10/core/ScalarType.h

test/test_torch.py

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

test/test_torch.py

ezyang · 2019-07-16T14:30:02Z

I'd kind of like to see some property-based tests for these. Maybe stuff like, if a floating point contains no significant information in the truncated part of the representation, you should get equivalent results adding floats and adding their corresponding bfloat16's.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

gchanan · 2019-08-05T21:00:57Z

test/test_torch.py

@@ -1707,6 +1707,11 @@ def test_add(self):
            expected = torch.zeros(2, 3, device=device).bool()
            self.assertEqual(res, expected)

+            # bfloat16
+            m1 = torch.tensor([1, 2], dtype=torch.bfloat16)


nit: wouldn't torch.tensor([1., 2.], dtype=torch.bfloat16) be the canonical representation?

gchanan · 2019-08-05T21:02:31Z

test/test_torch.py

+                self.assertRaises(RuntimeError, lambda: m1 - m2)
+            elif (dtype == torch.bfloat16):
+                # bfloat16 has a lower precision so we have to have a separate check for it
+                self.assertEqual(m1 - m2, torch.tensor([1.1016, 2.1094], dtype=dtype))


I still don't understand why we do this instead of just assertClose or whatever with a different atol/rtol.

gchanan · 2019-08-05T21:02:52Z

test/test_torch.py

+            if device == 'cpu':
+                a1 = torch.tensor([0.1, 0.1], dtype=torch.bfloat16, device=device)
+                a2 = torch.tensor([1.1, 0.1], dtype=torch.bfloat16, device=device)
+                self.assertEqual(a1 * a2, torch.tensor([0.1089, 0.0099], dtype=torch.bfloat16, device=device))


same as above, also there should be a comment here.

gchanan · 2019-08-05T21:03:02Z

test/test_torch.py

@@ -1851,6 +1877,10 @@ def test_div(self):
            res2[i, 3] = res2[i, 3] / 2
        self.assertEqual(res1, res2)

+        a1 = torch.tensor([4.2, 6.2], dtype=torch.bfloat16)
+        a2 = torch.tensor([2., 2.], dtype=torch.bfloat16)
+        self.assertEqual(a1 / a2, torch.tensor([2.0938, 3.0938], dtype=torch.bfloat16))


same as above, also there should be a comment here.

gchanan

as above.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

gchanan · 2019-08-05T22:15:26Z

test/test_torch.py

+                self.assertRaises(RuntimeError, lambda: m1 - m2)
+            else:
+                # bfloat16 has a lower precision so we have to have specified precision
+                self.assertEqual(m1 - m2, torch.tensor([1.11, 2.11], dtype=dtype), 0.01)


umm, isn't this setting lower precision for all dtypes except half and bool?

gchanan · 2019-08-05T22:15:36Z

test/test_torch.py

+            if device == 'cpu':
+                a1 = torch.tensor([0.1, 0.1], dtype=torch.bfloat16, device=device)
+                a2 = torch.tensor([1.1, 0.1], dtype=torch.bfloat16, device=device)
+                self.assertEqual(a1 * a2, torch.tensor([0.11, 0.01], dtype=torch.bfloat16, device=device), 0.01)


no, because this test doesnt test all dtypes

gchanan · 2019-08-05T22:15:40Z

test/test_torch.py

@@ -1851,6 +1872,10 @@ def test_div(self):
            res2[i, 3] = res2[i, 3] / 2
        self.assertEqual(res1, res2)

+        a1 = torch.tensor([4.2, 6.2], dtype=torch.bfloat16)
+        a2 = torch.tensor([2., 2.], dtype=torch.bfloat16)
+        self.assertEqual(a1 / a2, torch.tensor([2.1, 3.1], dtype=torch.bfloat16), 0.1)


no, because this test doesnt test all dtypes

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@izdeby has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: Enable Add, sub, mul, and div on CPU for bfloat16 type. Tested via unit tests. Pull Request resolved: pytorch/pytorch#22851 Differential Revision: D16256757 Pulled By: izdeby fbshipit-source-id: 8b62f7581fc0ca0d2cff48ab40d877a9fcf70a5b

facebook-github-bot · 2019-08-08T22:09:32Z

@izdeby merged this pull request in 5b9f55f.

izdeby changed the title ~~[WIP] Bfloat16 math ops~~ [WIP] Enable Add, sub, mul, and div on CPU for bfloat16 type. Jul 14, 2019

facebook-github-bot reviewed Jul 14, 2019

View reviewed changes

pytorchbot added module: cpu CPU specific problem (e.g., perf, algorithm) module: internals Related to internal abstractions in c10 and ATen module: operators labels Jul 15, 2019

izdeby commented Jul 15, 2019

View reviewed changes

aten/src/ATen/cpu/vec256/vec256_base.h Show resolved Hide resolved

facebook-github-bot reviewed Jul 15, 2019

View reviewed changes

izdeby requested review from gchanan, VitalyFedyunin, ezyang and bddppq July 15, 2019 15:47

izdeby changed the title ~~[WIP] Enable Add, sub, mul, and div on CPU for bfloat16 type.~~ Enable Add, sub, mul, and div on CPU for bfloat16 type. Jul 15, 2019

izdeby commented Jul 15, 2019

View reviewed changes

aten/src/ATen/native/cpu/BinaryOpsKernel.cpp Outdated Show resolved Hide resolved

gchanan reviewed Jul 15, 2019

View reviewed changes

c10/core/ScalarType.h Outdated Show resolved Hide resolved

gchanan reviewed Jul 15, 2019

View reviewed changes

test/test_torch.py Outdated Show resolved Hide resolved

gchanan reviewed Jul 15, 2019

View reviewed changes

test/test_torch.py Outdated Show resolved Hide resolved

facebook-github-bot reviewed Jul 15, 2019

View reviewed changes

VitalyFedyunin reviewed Jul 15, 2019

View reviewed changes

aten/src/ATen/native/cpu/BinaryOpsKernel.cpp Outdated Show resolved Hide resolved

VitalyFedyunin reviewed Jul 15, 2019

View reviewed changes

test/test_torch.py Show resolved Hide resolved

izdeby requested a review from gchanan July 15, 2019 20:13

gchanan requested changes Jul 15, 2019

View reviewed changes

c10/core/ScalarType.h Outdated Show resolved Hide resolved

test/test_torch.py Outdated Show resolved Hide resolved

test/test_torch.py Outdated Show resolved Hide resolved

izdeby force-pushed the bfloat16MathOps branch from 4507218 to 01fd8d7 Compare July 15, 2019 22:02

facebook-github-bot reviewed Jul 15, 2019

View reviewed changes

gchanan reviewed Jul 15, 2019

View reviewed changes

test/test_torch.py Show resolved Hide resolved

gchanan mentioned this pull request Jul 25, 2019

Port resize_as_ and clone from TH to Aten #23027

Closed

izdeby changed the title ~~Enable Add, sub, mul, and div on CPU for bfloat16 type.~~ [WIP] Enable Add, sub, mul, and div on CPU for bfloat16 type. Jul 31, 2019

izdeby force-pushed the bfloat16MathOps branch from 01fd8d7 to 9c54ae1 Compare July 31, 2019 21:15

izdeby requested review from gchanan and VitalyFedyunin July 31, 2019 21:15

izdeby force-pushed the bfloat16MathOps branch from d33c2fd to d48b2c2 Compare August 5, 2019 19:27

izdeby requested a review from ezyang August 5, 2019 19:27

facebook-github-bot reviewed Aug 5, 2019

View reviewed changes

gchanan reviewed Aug 5, 2019

View reviewed changes

gchanan requested changes Aug 5, 2019

View reviewed changes

izdeby requested a review from gchanan August 5, 2019 22:06

facebook-github-bot reviewed Aug 5, 2019

View reviewed changes

gchanan reviewed Aug 5, 2019

View reviewed changes

izdeby requested a review from gchanan August 5, 2019 22:35

facebook-github-bot reviewed Aug 5, 2019

View reviewed changes

ezyang removed their request for review August 6, 2019 13:28

gchanan approved these changes Aug 6, 2019

View reviewed changes

izdeby added 3 commits August 7, 2019 07:31

Enabled add, sub, mul, and div on CPU for bfloat16

64a506e

Updated tests with precision

6b94bab

Updated tests

0a85a96

izdeby force-pushed the bfloat16MathOps branch from af8f78e to 0a85a96 Compare August 7, 2019 14:32

facebook-github-bot reviewed Aug 7, 2019

View reviewed changes

facebook-github-bot closed this in 5b9f55f Aug 8, 2019

facebook-github-bot added the merged label Aug 8, 2019

facebook-github-bot deleted the bfloat16MathOps branch July 13, 2020 17:53

mruberry added the Merged label Oct 28, 2020

Enable Add, sub, mul, and div on CPU for bfloat16 type. #22851

Enable Add, sub, mul, and div on CPU for bfloat16 type. #22851

Uh oh!

Conversation

izdeby commented Jul 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gchanan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ezyang commented Jul 16, 2019

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchanan left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Aug 8, 2019

Uh oh!

Uh oh!

izdeby commented Jul 14, 2019 •

edited

Loading