Norm of complex Tensor

[HamedHojatian]

🐛 Bug

The norm function (torch.norm) doesn't work on the complex tensor in the pytorch version 1.7.1 . However, in version 1.6.0, it works.

To Reproduce

Steps to reproduce the behavior:

1. Example: torch.norm(torch.tensor([1+1j, 2+2j]))

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-28-872fa6766b44> in <module>
----> 1 x = torch.norm(torch.tensor([1+1j, 2+2j]))
      2 x

/export/tmp/hojatian/anaconda3/lib/python3.7/site-packages/torch/functional.py in norm(input, p, dim, keepdim, out, dtype)
   1288         if isinstance(p, str):
   1289             if p == "fro":
-> 1290                 return _VF.frobenius_norm(input, dim=(), keepdim=keepdim)  # type: ignore
   1291         if not isinstance(p, str):
   1292             _dim = [i for i in range(ndim)]  # noqa: C416 TODO: rewrite as list(range(m))
   
RuntimeError: frobenius norm not supported for complex tensors

Environment

• PyTorch Version: 1.7.1
• OS: Linux
• How you installed PyTorch: Conda
• Build command you used (if compiling from source): -
• Python version: 3.7.6
• CUDA/cuDNN version: cuda9.2.148_cudnn7.6.3_0
• GPU models and configuration: Tesla P100
• Any other relevant information: -

cc @ezyang @anjali411 @dylanbespalko @mruberry @jianyuh @nikitaved @pearu @heitorschueroff @walterddr @IvanYashchuk

[glaringlee]

@HamedHojatian
It works fine with the latest code built from master branch though.

[anjali411]

replica of #47833 (comment) @HamedHojatian this was fixed in #48284 and works on the latest master and will be added in the 1.8 release. let us know if you run into any issues with this function on the latest master. thanks!!

[mruberry]

cc @kurtamohler, this is fixed, right?

Thank you for reporting this issue, @HamedHojatian. We've been updating torch.norm's complex behavior recently, and in PyTorch 1.7 there's the NumPy-compatible torch.linalg.norm,

I'm surprised this worked in PyTorch 1.6. The tensor provided in the snippet is only 1D, and the frobenius norm only operates on matrices. I don't think the 2-norm support complex inputs, either, in PyTorch 1.6.

What did this produce in PyTorch 1.6?

[HamedHojatian]

@HamedHojatian
It works fine with the latest code built from master branch though.

Great! thank you ! Is cuda supporting it?

[HamedHojatian]

cc @kurtamohler, this is fixed, right?

Thank you for reporting this issue, @HamedHojatian. We've been updating torch.norm's complex behavior recently, and in PyTorch 1.7 there's the NumPy-compatible torch.linalg.norm,

I'm surprised this worked in PyTorch 1.6. The tensor provided in the snippet is only 1D, and the frobenius norm only operates on matrices. I don't think the 2-norm support complex inputs, either, in PyTorch 1.6.

What did this produce in PyTorch 1.6?

Thank you. As I check, the norm function is working correctly in 2D at 1.6.

[mruberry]

Interesting. Good thing it's fixed in PyTorch 1.8! Thanks for following up, @HamedHojatian!

[kurtamohler]

torch.norm() of order 2 for complex vector inputs was incorrect in 1.6, and inconsistent with the other supported vector norm orders, so it was purposely disabled for 1.7.

According to https://mathworld.wolfram.com/L2-Norm.html and numpy, the order 2 norm of a complex vector is vector.abs().pow(2).sum().pow(1/2). However, in pytorch 1.6, vector.pow(2).sum().pow(1/2) was being calculated instead. The absolute value (or modulus) of each element should be calculated first. In pytorch 1.6, we get, the following, showing that the absolute value of elements is not being calculated first.

>>> a = torch.tensor([1+1j, 2+2j])
>>> a.norm()
tensor(2.2361+2.2361j)
>>> a.pow(2).sum().pow(1/2)
tensor(2.2361+2.2361j)
>>> a.abs().pow(2).sum().pow(1/2)
tensor(3.1623)

This is what numpy gives, disagreeing with pytorch 1.6.

>>> numpy.linalg.norm([1+1j, 2+2j])
3.1622776601683795

And in pytorch 1.6, other vector norms calculate the absolute value of elements first with vector.abs().pow(order).sum().pow(1/order), not vector.pow(order).sum().pow(1/order). For p=3.2, for instance:

>>> a = torch.tensor([1+1j, 2+2j])
>>> a.norm(p=3.2)
tensor(2.9212+0.j)
>>> a.abs().pow(3.2).sum().pow(1/3.2)
tensor(2.9212)
>>> a.pow(3.2).sum().pow(1/3.2)
tensor(2.0656+2.0656j)

Pytorch 1.8 updates torch.norm to use the correct formula for order 2 with a complex vector.

[HamedHojatian]

torch.norm() of order 2 for complex vector inputs was incorrect in 1.6, and inconsistent with the other supported vector norm orders, so it was purposely disabled for 1.7.

According to https://mathworld.wolfram.com/L2-Norm.html and numpy, the order 2 norm of a complex vector is vector.abs().pow(2).sum().pow(1/2). However, in pytorch 1.6, vector.pow(2).sum().pow(1/2) was being calculated instead. The absolute value (or modulus) of each element should be calculated first. In pytorch 1.6, we get, the following, showing that the absolute value of elements is not being calculated first.

>>> a = torch.tensor([1+1j, 2+2j])
>>> a.norm()
tensor(2.2361+2.2361j)
>>> a.pow(2).sum().pow(1/2)
tensor(2.2361+2.2361j)
>>> a.abs().pow(2).sum().pow(1/2)
tensor(3.1623)

This is what numpy gives, disagreeing with pytorch 1.6.

>>> numpy.linalg.norm([1+1j, 2+2j])
3.1622776601683795

And in pytorch 1.6, other vector norms calculate the absolute value of elements first with vector.abs().pow(order).sum().pow(1/order), not vector.pow(order).sum().pow(1/order). For p=3.2, for instance:

>>> a = torch.tensor([1+1j, 2+2j])
>>> a.norm(p=3.2)
tensor(2.9212+0.j)
>>> a.abs().pow(3.2).sum().pow(1/3.2)
tensor(2.9212)
>>> a.pow(3.2).sum().pow(1/3.2)
tensor(2.0656+2.0656j)

Pytorch 1.8 updates torch.norm to use the correct formula for order 2 with a complex vector.

You are right. It is wrong. Thank you !