Implement more matrix functions #5840

Open
jiahao opened this Issue Feb 17, 2014 · 40 comments

Comments

Projects
None yet
Member

jiahao commented Feb 17, 2014 edited by stevengj

Higham and Deadman have published a list of software implementing algorithms for matrix functions, showing that Julia's library for these functions could be improved greatly relative to the state of the art.

The purpose of this issue is to discuss and track the implementation of matrix functions.

From Higham and Deadman's list:

General function evaluations

  • General matrix function with derivatives of the underlying scalar function available: Schur–Parlett algorithm (Davies and Higham, 2003)
  • Function of a symmetric or Hermitian matrix by diagonalization
  • Condition number estimate for general functions: cond(f,A)
  • matrix function - vector product evaluation: f(A) * b

Specific functions

  • expm: scaling and squaring algorithm (Al-Mohy and Higham, 2009)
  • logm: inverse scaling and squaring algorithm (Al-Mohy, Higham, and Relton, 2012,
    2013)
  • sqrtm:
    • Schur algorithm (Björck and Hammarling, 1983)
    • real version for real matrices (Higham, 1987)
    • blocking algorithm for performance improvements (Deadman, Higham, and Ralha, 2013)
  • A^t for real matrix powers: Schur– Padé algorithm (Higham and Lin, 2013) (#21184)
  • Matrix unwinding function

Fréchet derivatives

  • Exponential: scaling and squaring algorithm (Al-Mohy and Higham, 2009)
  • Logarithm: inverse scaling and squaring algorithm (Al-Mohy, Higham, and Relton, 2013)
  • General matrix function: complex step algorithm (Al-Mohy and Higham, 2010) or using block 2 × 2 matrix formula.
  • A^t for real matrix powers: Schur– Padé algorithm (Higham and Lin, 2013)

Miscellaneous functions

  • inv and det for Tridiagonal and SymTridiagonal matrices (Usmani, 1994) (#5358)
Owner

StefanKarpinski commented Feb 17, 2014

It would be amazing to include all of these algorithms. Thanks for putting this issue together, @jiahao.

Member

jiahao commented Feb 18, 2014

It's not clear to me what interface should be implemented for the very generic things like "function of a symmetric or Hermitian matrix".

We could define methods for elementary functions until the cows come home, but that is both tedious and namespace polluting (we'd need sinm, cosm, atanhm, etc. to disambiguate from the ordinary forms which broadcast over the elements) , and furthermore wouldn't scale well for linear combinations of functions.

Maybe we want something like applym, so that a call like applym(A->exp(A)+sin(2A)-cosh(4A), A) would evaluate the matrix expm(A)+sinm(2A)-coshm(4A)?

Owner

StefanKarpinski commented Feb 18, 2014

This is one of the reasons why having all these functions be vectorized is kind of shitty. I suspect that @JeffBezanson will vehemently agree. There could be a MatrixFunctions module that you have to import from to get the matrix variants with all the same names.

Member

jiahao commented Feb 18, 2014

Sure, but that won't help with the fact that a complicated matrix function like A->expm(A)+sinm(2A)-coshm(4A) can be computed more quickly all at once than computing (expm(A))+(sinm(2A))-(coshm(4A)) by working out the individual elementary functions separately, since the underlying Schur-Parlett algorithm is essentially the same.

Owner

StefanKarpinski commented Feb 18, 2014

Yes, that's a very good point.

Owner

stevengj commented Feb 18, 2014

Trefethen's contour-integration method is also neat for mostly analytic functions (with known singularities), though I don't know how it compares to the state of the art (in performance and generality).

Member

jiahao commented Feb 18, 2014

@stevengj it looks like that paper is Ref. 32 of the Higham and Deadman report, and isn't currently available in any major package (although it has Matlab code).

Contributor

simonbyrne commented Feb 18, 2014

This seems like a good candidate for a GSOC project.

As far as an interface goes, I would suggest a macro

B = @matfun exp(A)+sin(2A)-cosh(4A)

A naive approach could simply replace the functions (e.g. exp -> expm), a more advanced approach could try to identify the common decompositions.

Contributor

GunnarFarneback commented Feb 18, 2014

The item "[sqrtm] real version for real matrices (Higham, 1987)" is checked. I can't see that algorithm in base. I made an attempt on it about a year ago but concluded that the complexity just wasn't worth it. I've put up the code as a gist if someone wants to have a look.

Member

jiahao commented Feb 18, 2014

Thanks for catching that. I must have ticked it by accident when writing up this issue.

Just tried to call logm, but wasn't implemented. Had to go to Mathematica

Contributor

fabianlischka commented Aug 9, 2014

Whoever tackles this issue might find this paper helpful for testing:
Higham, Deadman: Testing Matrix Function Algorithms Using Identities, March 2014

Owner

StefanKarpinski commented Aug 9, 2014

Great reference – thanks, @fabianlischka.

ViralBShah removed the feature label Feb 14, 2015

Contributor

mfasi commented Jul 26, 2015

The logm function has been added by pull request #12217.

Owner

ViralBShah commented Jul 26, 2015

Member

ararslan commented May 5, 2016 edited

Would sinm and cosm (matrix sine and cosine, respectively, see here) be useful? They're easily derived from expm. I can put together a PR if that sounds desirable.

higham commented May 6, 2016

Dear Alex,

On 06/05/2016 00:11, Alex Arslan wrote:

Would |sinm| and |cosm| be useful? They're easily derived from |expm|. I can
put together a PR if that sounds desirable.

They are worth having.
It's possible to compute cosm(A) as (X+inv(X))/2, where X = expm(i*A), but
this requires complex arithmetic even when A is real, so is expensive.
It would be better to use the algorithms in

Awad Al-Mohy, Nicholas Higham & Samuel Relton, New Algorithms for
Computing the Matrix Sine and Cosine Separately or Simultaneously,
SIAM J. Sci. Comput. 37, A456-A487, 2015

https://github.com/sdrelton/cosm_sinm

---Nick

Member

ararslan commented May 6, 2016

Hi Nick, thanks for the heads up, I'll look at that paper.

While I have you here, would you be willing to license the MATLAB code for the unwinding matrix under MIT so it can be ported to Julia? I wasn't able to find a license for the code that accompanies the paper.

higham commented May 7, 2016

Dear Alex,

On 06/05/2016 21:55, Alex Arslan wrote:

While I have you here, would you be willing to license the MATLAB code for the
unwinding matrix under MIT so it can be ported to Julia? I wasn't able to find
a license for the code that accompanies the paper.

Ah, yes - there's a reason why we haven't made that available in the usual
licensed way on GitHub: the code uses subfunction

function [mm,ind] = swapping(m)

copied directly from the MATLAB funm. Since it wasn't an option to write
another version of that subfunction we just made the code available from

http://eprints.ma.man.ac.uk/2094/

I notice that we made the original 2003 funm codes available at

http://www.maths.manchester.ac.uk/~higham/NAMF/

with a GPL license and there is an earlier version of swapping.m there.
I don't remember how that version compared with the one that eventually
appeared in funm. Perhaps you can work from that older version?
If you want to do so and the GPL license is a problem I can change the
license on the above page.

---Nick

Member

ararslan commented May 7, 2016

Please correct me if I'm wrong, but my understanding is that MIT and GPL are incompatible; GPL is a much more stringent license that requires that derived works, which includes direct ports of the code, be licensed under GPL as well. Further, if one portion of a project is licensed under GPL, the rest of the project must also be GPL. Since Julia's core is entirely MIT, that wouldn't work; the port would have to be its own GPL-licensed package. So if you're willing to license any of your code under MIT to facilitate porting to Julia, that would be fantastic!

Owner

Keno commented May 7, 2016

The MIT and the GPL are not incompatible. The result however, would be GPL licensed as a whole distribution. Since the restrictions of the GPL are too stringent for the standard library of a programming language however, we generally do not accept new contributions of GPL licensed code into base julia and are actively trying to move existing GPL licensed code out of base (there's already a makefile flag that disables such code).

higham commented May 8, 2016

On 07/05/2016 19:56, Alex Arslan wrote:

So if you're willing to license any of your code under MIT to facilitate
porting to Julia, that would be fantastic!

Done - the page

http://www.maths.manchester.ac.uk/~higham/NAMF/

has been updated with the MIT license.

---Nick

Owner

ViralBShah commented May 8, 2016

@higham Thank you.

Member

ararslan commented May 8, 2016

@higham That's fantastic, thank you very much!

Owner

stevengj commented Aug 2, 2016

Note that we are planning on deprecating all of the elementwise "vectorized" operations like exp(A) in favor of exp.(A) in 0.6 (see #17302). So, at some point in the future we may be able to reclaim those names to use for matrix functions.

👍🏽

Member

ararslan commented Aug 2, 2016

@stevengj I thought the convention for matrix functions was to append m, e.g. expm?

Owner

stevengj commented Aug 2, 2016

@ararslan, that's a Matlabism that is imposed on the language because exp(A) is elementwise; the underlying mathematical matrix function has no "m" suffix in the math literature. See #8450 for a long discussion of why/how we moved away from implicitly vectorized functions like exp(A) in Julia.

Member

ararslan commented Aug 2, 2016

I've read through there, I just didn't realize there was any desire to do away with the m carryover from Matlab.

Owner

stevengj commented Aug 2, 2016

I don't think we should rush into ditching the m suffix; it's just something we can consider in the future. We will want to have exp(A) give a deprecation message to use exp.(A) for quite a while, to teach users about the dot syntax.

I think dropping the m is more consistent with the Taylor series definition of exp. For example,

exp(dual(1.,2.)) 

returns the dual corresponding to the Taylor series definition (which in this case is equivalent to expm([1. 2.; 0. 1.]).

That's ignoring the Matlab and Mathematica precedents.

Whether or not exp(A) replaces expm(A), I think down the line exp.(A) should replace exp(A), as the current exp(A) is confusing for non-MATLAB users

Contributor

felixrehren commented Jan 11, 2017 edited

Version 2 of the Deadman-Higham list has been out for a year and Julia is the same as it was first time. Let's get better before version 3 comes out. I'm going to look at Schur-Pade -- that together with the top-4 tasks "General function evaluations" in the OP would cover most of this. Anyone else want to join?

P.S. we already beat R in the DH list, with built-in Matlab and SciPy looking vulnerable as well!

Contributor

simonbyrne commented Jan 11, 2017 edited

Is the "Deadman-Higham ranking" our Weissman Score?

Could there be a function matrixbroadcast(f,A) that calculates the matrix function f(A) for general f? This would be easy to implement for diagonalisable A, not sure about otherwise

Owner

stevengj commented Jan 12, 2017 edited

Even for diagonalizable A, the possibility of nearly defective A would be an issue. Of course, it could be restricted to Hermitian A, but that case is so easy that we would hardly add much value by implementing a function for it in Base.

Contributor

alanedelman commented Feb 1, 2017

There is a pretty good chance that matrix derivatives would work with ForwardDiff.jl, at least
if there is a corresponding generic linear algebra function. I've been pretty impressed that
one can get derivatives of powers and svd's and inverses, etc.

Contributor

ChrisRackauckas commented Feb 22, 2017

Should this all be moved out to a MatrixFunctions.jl?

Contributor

felixrehren commented Feb 23, 2017

Seems like a good idea -- I think everything on the list other than expm logm sqrtm is a clear candidate for moving out (e.g. I have a rootm function that takes arbitrary matrix pth roots -- probably doesn't fit in base, but could be nice somewhere). Whether expm logm sqrtm are worth keeping in base or can move out as well, I don't know.

@ararslan You did the great work moving SpecialFunctions.jl. What do you think about MatrixFunctions.jl? Other people that were involved there IIRC but are not in this thread: @musm, @andreasnoack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment