Add scalar.conj() and update backward formulas for add and sub #46596

anjali411 · 2020-10-20T17:17:15Z

Added conj method for scalar similar to numpy.
Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex.
Enabled complex backward for nonzero (no formula update needed).

Stack from ghstack:

Add scalar.conj() and update backward formulas for add and sub #46596 Add scalar.conj() and update backward formulas for add and sub

Differential Revision: D24529227

[ghstack-poisoned]

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). [ghstack-poisoned]

dr-ci · 2020-10-20T17:37:25Z

💊 CI failures summary and remediations

As of commit 5b6d3a7 (more details on the Dr. CI page):

✅ None of the CI failures appear to be your fault 💚

1/1 tentatively recognized as flaky ❄️
- Click here to rerun these jobs

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

pytorch_linux_xenial_py3_clang5_android_ndk_r19c_vulkan_x86_32_build (1/1)

Step: "Build" (full log | diagnosis details | 🔁 rerun) ❄️

Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

Parallel backend flags:  
DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-xenial-py3-clang5-android-ndk-r19c:4d22616f4e3faf830d069137acd2208e7a023625 
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 19 times.

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). [ghstack-poisoned]

ghstack-source-id: 422aeb2ba1dcdc9d526e2c9b1b73996bbd64831b Pull Request resolved: #46596

ezyang · 2020-10-21T19:40:19Z

c10/core/Scalar.cpp

+  if (isComplex()) {
+    return Scalar(std::conj(v.z));
+  } else if (isFloatingPoint()) {
+    return Scalar(v.d);


can't you just return *this here?

ezyang

At some point when you're not in a rush, it would be good to use @robieta's tools for instruction counts to assess what the non-complex overheads of the new complex idioms and conjugations are. It will be a little tricky to setup since you have to go through autograd, but we might be able to setup a microbenchmark where we just call the microbenchmark directly. That will help us understand how urgent extra optimizing here is (which I know @ngimel is nervous about the conj additions here.)

mruberry

Test changes look good. Test failure doesn't appear to be related.

aten/src/ATen/test/scalar_test.cpp

tools/autograd/gen_variable_type.py

gchanan · 2020-10-21T22:25:28Z

tools/autograd/derivatives.yaml

@@ -995,11 +995,11 @@
  self: std_backward(result, grad, self, dim, unbiased, keepdim)

 - name: sub.Tensor(Tensor self, Tensor other, *, Scalar alpha=1) -> Tensor
-  self: grad
-  other: -grad * alpha
+  self: handle_r_to_c(self.scalar_type(), grad)


why do we need this? This seems pretty similar to

pytorch/torch/csrc/autograd/engine.cpp

Line 611 in e02a3e1

grad = grad.to(c10::typeMetaToScalarType(metadata.options().dtype()));

-- is it because to doesn't work c->r?

.to works for C->R. currently handle_r_to_c returns a (real) view of the final gradient value in case the gradient is complex and the input is not.

anjali411 · 2020-10-22T02:03:30Z

At some point when you're not in a rush, it would be good to use @robieta's tools for instruction counts to assess what the non-complex overheads of the new complex idioms and conjugations are. It will be a little tricky to setup since you have to go through autograd, but we might be able to setup a microbenchmark where we just call the microbenchmark directly. That will help us understand how urgent extra optimizing here is (which I know @ngimel is nervous about the conj additions here.)

I think that's a great idea! is there a doc to help with the setup?

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). [ghstack-poisoned]

ghstack-source-id: 3fb369be52893955af285a321c874c210d34aa83 Pull Request resolved: #46596

ezyang · 2020-10-23T02:26:19Z

Hopefully Taylor's talk today should help? He's also been working on written docs as well.

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: a94571f9f3745d984c3ec8cf2b7c6eb829acfb36 Pull Request resolved: #46596

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: 1ddd2625999bcd6b02ea6c6c393f16c24d859f31 Pull Request resolved: #46596

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: 459ce186587e2bff4acb8dae8ce3728333c918c3 Pull Request resolved: #46596

… sub" 1. Added `conj` method for scalar similar to numpy. 2. Updates backward formulas for add and sub to work correctly for R -> C cases and for the case when alpha is complex. 3. Enabled complex backward for nonzero (no formula update needed). Differential Revision: [D24529227](https://our.internmc.facebook.com/intern/diff/D24529227) [ghstack-poisoned]

ghstack-source-id: 9a8402e75da9dfdbeb9dfe77a49f95f9a99999ae Pull Request resolved: #46596

facebook-github-bot · 2020-11-03T01:13:36Z

@anjali411 merged this pull request in cedeee2.

albanD · 2021-03-02T21:16:50Z

torch/testing/_internal/common_methods_invocations.py

@@ -538,6 +538,7 @@ def method_tests():
        ('add', (), ((S, S, S),), 'scalar_broadcast_lhs', (True,)),
        ('add', (S, S, S), (3.14,), 'constant', (True,)),
        ('add', (), (3.14,), 'scalar_constant', (True,)),
+        ('add', (S, S, S), (3.14j,), 'complex_scalar_constant', (True,)),


Does this actually calls into the scalar overload? I think that the python binding is converting these to Tensors all the time.

You can double check this by doing:

import torch a = torch.rand(10, requires_grad=True) print((a + a).grad_fn) print((a + 1.).grad_fn)

That will print

<AddBackward0 object at 0x7fc1c1446150> <AddBackward0 object at 0x7fc1c2f14f50>

The important part is the 0 here that means that both are actually using the 0th overload (the one for Tensor inputs).

Add scalar.conj() and update backward formulas for add and sub

b6be40c

[ghstack-poisoned]

anjali411 requested review from albanD and gchanan October 20, 2020 17:17

anjali411 added complex_autograd module: complex Related to complex number support in PyTorch labels Oct 20, 2020

anjali411 requested a review from ezyang October 20, 2020 17:22

anjali411 added a commit that referenced this pull request Oct 20, 2020

Add scalar.conj() and update backward formulas for add and sub

79cb7e2

ghstack-source-id: 422aeb2ba1dcdc9d526e2c9b1b73996bbd64831b Pull Request resolved: #46596

anjali411 mentioned this pull request Oct 20, 2020

Make torch.det() support complex input. #45980

Closed

anjali411 requested review from zou3519 and mruberry October 21, 2020 00:45

ezyang reviewed Oct 21, 2020

View reviewed changes

ezyang approved these changes Oct 21, 2020

View reviewed changes

mruberry approved these changes Oct 21, 2020

View reviewed changes

gchanan reviewed Oct 21, 2020

View reviewed changes

aten/src/ATen/test/scalar_test.cpp Outdated Show resolved Hide resolved

gchanan reviewed Oct 21, 2020

View reviewed changes

tools/autograd/gen_variable_type.py Outdated Show resolved Hide resolved

gchanan reviewed Oct 21, 2020

View reviewed changes

anjali411 added a commit that referenced this pull request Oct 22, 2020

Add scalar.conj() and update backward formulas for add and sub

758e59a

ghstack-source-id: 3fb369be52893955af285a321c874c210d34aa83 Pull Request resolved: #46596

anjali411 added a commit that referenced this pull request Oct 26, 2020

Add scalar.conj() and update backward formulas for add and sub

30359fc

ghstack-source-id: a94571f9f3745d984c3ec8cf2b7c6eb829acfb36 Pull Request resolved: #46596

anjali411 added a commit that referenced this pull request Oct 27, 2020

Add scalar.conj() and update backward formulas for add and sub

f00ab65

ghstack-source-id: 1ddd2625999bcd6b02ea6c6c393f16c24d859f31 Pull Request resolved: #46596

anjali411 added a commit that referenced this pull request Oct 28, 2020

Add scalar.conj() and update backward formulas for add and sub

633c8a6

ghstack-source-id: 459ce186587e2bff4acb8dae8ce3728333c918c3 Pull Request resolved: #46596

facebook-github-bot added the cla signed label Oct 30, 2020

anjali411 added a commit that referenced this pull request Nov 2, 2020

Add scalar.conj() and update backward formulas for add and sub

f73ffc9

ghstack-source-id: 9a8402e75da9dfdbeb9dfe77a49f95f9a99999ae Pull Request resolved: #46596

facebook-github-bot closed this in cedeee2 Nov 3, 2020

facebook-github-bot added the Merged label Nov 3, 2020

facebook-github-bot deleted the gh/anjali411/66/head branch November 6, 2020 15:19

albanD reviewed Mar 2, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scalar.conj() and update backward formulas for add and sub #46596

Add scalar.conj() and update backward formulas for add and sub #46596

anjali411 commented Oct 20, 2020 •

edited

dr-ci bot commented Oct 20, 2020 •

edited

ezyang Oct 21, 2020

ezyang left a comment

mruberry left a comment

gchanan Oct 21, 2020

anjali411 Oct 22, 2020

anjali411 commented Oct 22, 2020 •

edited

ezyang commented Oct 23, 2020

facebook-github-bot commented Nov 3, 2020

albanD Mar 2, 2021

Add scalar.conj() and update backward formulas for add and sub #46596

Add scalar.conj() and update backward formulas for add and sub #46596

Conversation

anjali411 commented Oct 20, 2020 • edited

dr-ci bot commented Oct 20, 2020 • edited

💊 CI failures summary and remediations

❄️ 1 failure tentatively classified as flaky

pytorch_linux_xenial_py3_clang5_android_ndk_r19c_vulkan_x86_32_build (1/1)

ezyang Oct 21, 2020

Choose a reason for hiding this comment

ezyang left a comment

Choose a reason for hiding this comment

mruberry left a comment

Choose a reason for hiding this comment

gchanan Oct 21, 2020

Choose a reason for hiding this comment

anjali411 Oct 22, 2020

Choose a reason for hiding this comment

anjali411 commented Oct 22, 2020 • edited

ezyang commented Oct 23, 2020

facebook-github-bot commented Nov 3, 2020

albanD Mar 2, 2021

Choose a reason for hiding this comment

anjali411 commented Oct 20, 2020 •

edited

dr-ci bot commented Oct 20, 2020 •

edited

anjali411 commented Oct 22, 2020 •

edited