[feature request] Add matrix functions

[carlosgmartin]

Add matrix power as implemented by numpy.linalg.matrix_power, matrix exponential as implemented by scipy.linalg.expm, and matrix logarithm as implemented by scipy.linalg.logm.

• matrix_power
• matrix_exp
• matrix_log (For an implementation see [feature request] Add matrix functions #9983 (comment))
• matrix_sqrt (For an implementation see [feature request] Add matrix functions #9983 (comment))

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @jianyuh @nikitaved @pearu @mruberry @heitorschueroff @walterddr @IvanYashchuk @xwang233 @lezcano @rgommers

[fmassa]

Matrix power can be easily implemented in PyTorch by following the numpy implementation, which translates straight away to PyTorch, see implementation here.
Matrix exponential seems a bit more involved though

[vishwakftw]

Are these worth porting into ATen? If so, I can take it up.

[zou3519]

Yes, please!

[vadimkantorov]

A generalization based on SVD (https://en.wikipedia.org/wiki/Matrix_function) may also be useful (for simple graph signal processing): diagonalize the matrix somehow (svd by default, maybe some optimized svd for symmetric matrices), apply the user-supplied function to eigenvalues, re-multiply the factors back.

For diagonalizable matrices, this way of doing matrix power may also be faster (for large powers). This would also enable matrix square root, though for matrix square root there seem to be specialized approx algorithms based on newton iteration.

[vishwakftw]

@zou3519 @fmassa I went through the SciPy implementation for matrixexp (named expm in SciPy), and have a few questions:

• the implementation is added for sparse arrays as well. Should I be considering that, or is that supposed to be left for later?
• the implementation seems to take advantage of upper triangular matrices, and there are some optimized calculations done for the same. Should I take the same approach with ATen or is a general implementation sufficient?
  Thank you.

[oleg-kachan]

Upvote for this, matrix exponential expm, logarithm logm and (fractional) matrix power powm which includes square root (with negative powers support)

[harpone]

The approach by @vadimkantorov seems to make most sense IMO because there's already a (differentiable) PyTorch function for SVD, and after that the exponentiation is just elemwise exp for the diagonal.

There's also an elegant (and probably useful) expression for the gradient. I don't actually know how the pytorch gradients are defined (yet), but I could probably dig up/work out the math for the gradient if there's interest for that...
EDIT: oops actually it's probably not useful in practice... anyway, the SVD + elemwise exp seems good

[ssnl]

Already implemented

[harpone]

OK great to hear that @ssnl! Can you point out the function (also for reference to anyone else that happens to stumble on this thread)? Can't find it in the docs...

[vishwakftw]

This issue is partially resolved: matrix_power is implemented, whereas matrix_exp is not yet done.

[fmassa]

@harpone here it is for reference https://pytorch.org/docs/master/torch.html?highlight=matrix_power#torch.matrix_power
Note that it is only available in the nightly build, not in 0.4.1