Support complex torch.solve on the GPU

[boeddeker]

🚀 Feature

Support torch.solve on the GPU

Motivation

In pytorch/audio#768 I asked about the complex support in torch for some linear algebra functions and
@anjali411 said, I should post here my feature request.

Pitch

It would be nice, when the following code would not fail in the last line:

>>> import numpy as np, torch
>>> torch.__version__
'1.7.0.dev20200705+cu101'
>>> t = torch.tensor(np.random.randn(2, 2))
>>> _ = torch.solve(t, t)
>>> _ = torch.solve(t.to(0), t.to(0))
>>> t = torch.tensor(1j * np.random.randn(2, 2))
>>> _ = torch.solve(t, t)
>>> _ = torch.solve(t.to(0), t.to(0))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: "solve_cuda" not implemented for 'ComplexDouble'

Alternatives

Currently, I am using
https://github.com/kamo-naoyuki/pytorch_complex/tree/master/torch_complex
but a native support would be faster, consume less memory and the code would contain fewer hacks to distinguish real and complex tensors.

Additional context

cc @ezyang @anjali411 @dylanbespalko

[mruberry]

Thanks for the suggestion, @boeddeker! I updated the title to clarify the request is for complex solve.

[anjali411]

@boeddeker Thanks for creating the issue! I agree that it would be super useful to have torch.solve supported for complex so that we won't need the hacks anymore. I or someone else will work on adding this once torch.matmul is supported for complex numbers (since it's used in the backwards).

[boeddeker]

Thanks, that would be great