Implement NumPy-like function torch.float_power() #44937

Kiyosora · 2020-09-18T06:36:23Z

Related with NumPy-like Functionality Request Rollup #38349
Implementing the NumPy-like function torch.float_power() .

dr-ci · 2020-09-18T09:44:33Z

💊 CI failures summary and remediations

As of commit 453287e (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 231 times.

Kiyosora · 2020-09-22T06:40:37Z

Hi, @mruberry ! A new NumPy-like function torch.float_power() is implemented in this PR.
Please kindly help to take a look at your convenience. 😃

mruberry · 2020-09-22T09:02:04Z

This is awesome, @Kiyosora! Just an fyi that we're finishing the PyTorch 1.7 release this week, so this review may be delayed a few days. Sorry for not being more responsive.

Kiyosora · 2020-09-22T09:50:53Z

Thanks for the rapid reply, @mruberry !
I wouldn't mind for waiting, and hope everything goes well for the PyTorch 1.7 release. 🎉

codecov · 2020-09-25T16:59:34Z

Codecov Report

Merging #44937 (453287e) into master (18ae12a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #44937   +/-   ##
=======================================
  Coverage   80.92%   80.93%           
=======================================
  Files        1855     1855           
  Lines      200156   200181   +25     
=======================================
+ Hits       161981   162011   +30     
+ Misses      38175    38170    -5

aten/src/ATen/native/Pow.cpp

mruberry · 2020-10-01T07:26:47Z

aten/src/ATen/native/Pow.cpp

+  TORCH_CHECK(result.scalar_type() == dtype,
+              "output type ", result.scalar_type(), "is not the desired output type ", dtype);
+
+  if (exp.isComplex() && (exp.toComplexDouble() == 0.0) ) {


resize_output to resize instead of resize_as. (pow also needs to be updated, but it doesn't need to be updated in this PR.)

Progressing in the separate PR #46830

mruberry · 2020-10-01T07:26:57Z

aten/src/ATen/native/Pow.cpp

+  if (exp.isComplex() && (exp.toComplexDouble() == 0.0) ) {
+    result.resize_as_(base).fill_(1);
+  } else if (exp.isComplex() && (exp.toComplexDouble() == 1.0) ) {
+    result.resize_as_(base).fill_(base);


Would you please fix this for pow, too, actually. Sample code demonstrating the error:

torch.tensor((1 + 1j, 2 + 2j)) ** complex(1, 0) : RuntimeError: fill_ only supports 0-dimension value tensor but got tensor with 1 dimensions.

Progressing as a separate PR #46830

aten/src/ATen/native/Pow.cpp

mruberry · 2020-10-01T07:45:16Z

aten/src/ATen/native/native_functions.yaml

@@ -6201,6 +6201,35 @@
    CPU, CUDA: pow
    SparseCPU, SparseCUDA: pow_sparse_scalar

+- func: float_power.Tensor_Tensor_out(Tensor self, Tensor exponent, *, Tensor(a!) out) -> Tensor(a!)
+  dispatch:


Do these need explicit dispatches? I think the system will work out which functions to call. Explicitly defining the dispatch will, perhaps surprisingly, prevent float_power from deriving its derivative if it's implemented as a composite operation (as suggested above).

pow also has an inplace variant, pow_:

pytorch/aten/src/ATen/native/native_functions.yaml

Line 5031 in 4339f5c

- func: pow_.Scalar(Tensor(a!) self, Scalar exponent) -> Tensor(a!)

Those entries should probably be moved next to the other pow entries, too.

Once I remove the explicit dispatches, the following error comes out:

AssertionError: There's a formula for float_power(or its functional variant) in derivatives.yaml. It's required to add a dispatch section for it with explicit supported backends e.g CPU/CUDA or DefaultBackend in native_functions.yaml. Please see https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native#choosing-the-right-dispatch-keyword for instructions to choose the right dispatch keyword.

So, I guess we need the explicit dispatches here for autograd.

In addition, The improvement for pow_ is now progressing in the separate PR #46830.

The way dispatch happens has actually changed. I think the correct dispatch for these is now Math: instead of CPU, CUDA since float_power is implemented using pow.

It seems that when using Math: as dispatch, both autograd test and XLA test will suffer from the precision lack. Even if I directly call pow without any dtype casting like this below, the problem still exists.

Tensor float_power(const Tensor& base, const Tensor& exp) { return at::pow(base, exp); }

Since using CPU, CUDA can avoid from precision lack, maybe we should revert to it?
I am not familiar with autograd yet, maybe I have missed something... 😕

Sorry, can you elaborate on the lack of precision you're seeing, and how changing the dispatch can help with it?

Sorry for the late reply, @mruberry
When I use Math as dispatch, an assertEqual error occurs in test_autograd, saying that the gradients calculated by in-place variant is inconsistent with the general, just like:

>>> a=torch.tensor([1.,2.,3.], dtype=torch.double, requires_grad=True) >>> b=torch.tensor([1.,2.,3.], dtype=torch.double, requires_grad=True) >>> grad=torch.randn_like(a).double() >>> a.float_power(2.).backward(grad) >>> a.grad tensor([-4.0256, -1.6108, 1.2338], dtype=torch.float64) >>> b.float_power_(2.).backward(grad) >>> b.grad tensor([-6.0385, -2.0134, 1.4394], dtype=torch.float64)

But in fact, the in-place variants usually not allow to calculating gradients, the original pow is also doing as so.

>>> a.pow_(2.).backward(grad) Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

And when I changed dispatch from Math to CPU, CUDA, making a define in tools/autograd/derivatives.yaml (as we do in the previous version), The above abnormal phenomenon was eliminated.
It seems that there still not have any in-place variant used Math as dispatch so far, so I doubt it may related with this phenomenon...

Thank you for this explanation, this is extremely interesting. cc @ailzhang and @albanD to take a look, too.

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.

This error is unrelated to the pow formula, it only happens because you modify your leaf inplace. Doing a.clone().pow_(2.) should work just fine.

saying that the gradients calculated by in-place variant is inconsistent with the general

If you don't provide a formula directly in derivatives.yaml, you need to make sure to only ever call functions that do from your generic aten implementation. In particular, always call the at:: version and not native:: version of ops.

docs/source/tensors.rst

torch/_torch_docs.py

This reverts commit f4b646eb8aa2db98ddb696036f59af26925e41d6.

mruberry · 2020-11-26T07:18:56Z

test/test_torch.py

+
+            # Exception case test in test_float_power_exceptions
+            if op is torch.Tensor.float_power_ and base_dtype != out_dtype:
+                continue


Instead of just continuing here, attempt to perform the operation and verify it throws an error using self.assertRaisesRegex.

mruberry · 2020-11-26T07:20:40Z

test/test_torch.py

+
+                # Exception case test in test_float_power_exceptions
+                if op is torch.Tensor.float_power_ and base_dtype != out_dtype_scalar_exp:
+                    continue


As above, verify this throws an error instead of continuing

mruberry · 2020-11-26T07:24:25Z

torch/_torch_docs.py

+If neither input is complex returns a ``torch.float64`` tensor, 
+and if one or more inputs is complex returns a ``torch.complex128`` tensor.
+
+:attr:`exponent` can be either a single number, or a `Tensor`


I would remove this paragraph. It's correct, but a little misleading since this PR also allows :attr:`input` to be a scalar. The type support can be mentioned below, too, instead of here.

mruberry · 2020-11-26T07:26:45Z

torch/_torch_docs.py

+:attr:`exponent` can be either a single number, or a `Tensor`
+with the same number of elements as :attr:`input`.
+
+.. math::


This mathematical portion is also correct but can be removed. The description above it is already very clear.

mruberry · 2020-11-26T07:27:32Z

torch/_torch_docs.py

+    like when an integer base is raised to a negative integer exponent.
+
+Args:
+    {input}


"{input}" -> "input (Tensor or Number): the base value(s)"

mruberry · 2020-11-26T07:27:40Z

torch/_torch_docs.py

+
+Args:
+    {input}
+    exponent (Tensor or Number): the exponent value


"value" -> "values"

mruberry

Hey @Kiyosora!

Sorry for the delay in reviewing this PR. This week is a holiday for the PyTorch team.

This PR is very good and close to being ready. I made a few small final suggestions. Looking forward to reviewing the final update!

Kiyosora · 2020-11-26T15:09:06Z

Hey @Kiyosora!

Sorry for the delay in reviewing this PR. This week is a holiday for the PyTorch team.

This PR is very good and close to being ready. I made a few small final suggestions. Looking forward to reviewing the final update!

Happy Thanksgiving @mruberry !

Really appreciate for your consistant help!

I updated this PR by you suggestions,
Please take a look in any convenient time after your vacation. 😃

mruberry

Nice work! Thanks @Kiyosora!

facebook-github-bot

@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-11-28T03:09:23Z

@mruberry merged this pull request in 272f4db.

Summary: - Related with #44937 - Use `resize_output` instead of `resize_as` - Tuning the `native_functions.yaml`, move the inplace variant `pow_` next to the other `pow` entries Pull Request resolved: #46830 Reviewed By: mrshenli Differential Revision: D24567702 Pulled By: anjali411 fbshipit-source-id: a352422c9d4e356574dbfdf21fb57f7ca7c6075d

pytorchbot added the open source label Sep 18, 2020

Kiyosora force-pushed the implement_float_power branch from b4b6ae7 to 3d60565 Compare September 18, 2020 07:59

Kiyosora force-pushed the implement_float_power branch 5 times, most recently from 4c75c49 to 6f5552f Compare September 21, 2020 09:05

Kiyosora changed the title ~~[WIP] Implement NumPy-like function torch.float_power()~~ Implement NumPy-like function torch.float_power() Sep 21, 2020

Kiyosora marked this pull request as ready for review September 21, 2020 15:00

Kiyosora force-pushed the implement_float_power branch from 6f5552f to a98486e Compare September 22, 2020 01:56

ailzhang requested a review from mruberry September 22, 2020 19:59

ailzhang added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Sep 22, 2020

Kiyosora force-pushed the implement_float_power branch from a98486e to bd88b41 Compare September 25, 2020 09:37