feature request: differentiable matrix solve

[dbbert]

Are there any plans to implement something like tensorflow's tf.matrix_solve() and tf.matrix_solve_ls()?

torch.gels() exists but it doesn't have the backward step implemented.

[fmassa]

One workaround for the moment would be to write matrix_solve as x = A.inverse() * b, which should support backprop since #1670 .
Adding support for matrix_solve should be trivial, the gradient for it can be found in https://github.com/tensorflow/tensorflow/blob/359d6f9716c0bb9bd8201ce600da98b0481a8049/tensorflow/python/ops/linalg_grad.py#L81-L91

[fmassa]

I'll add backward support for gesv, which should be equivalent to tf.matrix_solve. I'll try to do it later today.

[MarcoForte]

How difficult would it be to add in batched versions (where only the final two dimensions are square) of GESV and inverse? Are there any plans for this?

[fmassa]

If we have batched version of gesv, then adapting the autograd part is very simple.
There is an open issue already that discuss implementing batched gesv #3164

[fmassa]

Actually, there is an open PR for that, so it might be coming soon #6100

[zou3519]

I'll rebase the batched version of gesv. It'll come in soon

[D-X-Y]

@fmassa May I ask why gesv only support A is a square matrix. As I saw from https://www.tensorflow.org/api_docs/python/tf/matrix_solve_ls, it support that A is a MxN matrix, and B is a NxK matrix, then solve a matrix X (NxK). If I want to use a non-square matrix A in gesv, would you mind to let me know how to implement in PyTorch?

[zou3519]

@D-X-Y this doesn't exactly answer your question but you can add zero-rows to the A matrix until it becomes square.

[D-X-Y]

@zou3519 thanks a lot, I find the solution, torch.gels can solve the problem.

[bobchennan]

Any plan to add gradient calculation for gels?

[LemonPi]

I am also wondering if torch.gels will have differentiation support.
My current workaround is to use torch.mm(X.pinverse(), Y) instead

[bobchennan]

https://gist.github.com/bobchennan/a865b153c6835a3a6a5c628213766150 I actually implemented overdetermined case following tensorflow implementation. For backward pass I only did basic grad test.

…

On Monday, March 18, 2019, Johnson Zhong ***@***.***> wrote: I am also wondering if torch.gels will have differentiation support. My current workaround is to use torch.mm(X.pinverse(), Y) instead — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1724 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA5UUNUEgeqv37Zxc01iV6hW17gScRQfks5vYADxgaJpZM4NwDQO> .

[LemonPi]

https://gist.github.com/bobchennan/a865b153c6835a3a6a5c628213766150 I actually implemented overdetermined case following tensorflow implementation. For backward pass I only did basic grad test.
…
On Monday, March 18, 2019, Johnson Zhong @.***> wrote: I am also wondering if torch.gels will have differentiation support. My current workaround is to use torch.mm(X.pinverse(), Y) instead — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1724 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5UUNUEgeqv37Zxc01iV6hW17gScRQfks5vYADxgaJpZM4NwDQO .

Thanks, seems to work fine!