WIP: Add specialized methods for inverses of small matrices

[KristofferC]

Same as #12452 but for inverses

The following benchmark shows the difference between master and PR (using #12452):

function f(N, n)
    A = rand(n, n)
    @time for i = 1:N
        inv(A)
    end
end

Master

f(10^5, 2)
#  0.094840 seconds (800.00 k allocations: 48.828 MB, 5.10% gc time) 
f(10^5, 3)
#  0.124090 seconds (800.00 k allocations: 59.509 MB, 4.77% gc time)

PR

f(10^5, 2);
#0.009085 seconds (100.00 k allocations: 9.155 MB)
f(10^5, 3);
#0.012562 seconds (100.00 k allocations: 13.733 MB)

[mlubin]

Is exact comparison with zero reasonable here?

[KristofferC]

Probably not. I thought about it but I didn't know what would be reasonable due to the generic T. eps(T) maybe?

[andreasnoack]

I think it is fine to throw for exact zeros only. That is similar to what LAPACK does.

[andreasnoack]

These functions actually assume commutativity for multiplication of the elements so the elements should probably be restricted to subtypes of Union{Real,Complex}.

[KristofferC]

I restricted the types for when the small_inv method is called now.

[KristofferC]

Could someone restart the AppVeyor build for me?

[tkelman]

You need to rebase and force push here, you hit a build-number collision appveyor/ci#353

[KristofferC]

To note, the timings here are with the hand coded determinants in #12452. These might be unacceptable due to numerical instability. WIth only #12460 the timings in this PR are:

f(10^5, 2);
#  0.045274 seconds (700.00 k allocations: 41.199 MB, 8.82% gc time)
f(10^5, 3);
#  0.052139 seconds (700.00 k allocations: 51.880 MB, 6.88% gc time)

which is only about a factor 2x faster.

[StefanKarpinski]

This looks pretty reasonable. @andreasnoack, please review and merge if you think this is good to go.

[andreasnoack]

There are two issues here.

1. I think most of the speed up will depend on the faster determinant in WIP: Add specialized methods for determinant of small matrices #12452, and they have a larger error so I'd like to find another solution for them.
2. Even with more precise determinants, I think these inverses might be imprecise relative to inverses based on the LU. I'll have to run some tests to confirm that. @KristofferC have you looked into the errors?

[mlubin]

Couldn't you hand-code a 2x2 LU, for example, to work around the numerical stability issue?
That seems to be the motivation behind the proposal for the modified hand-coded determinant also (#12452 (comment)).

[andreasnoack]

@mlubin Sure. I think that is the way to procees, but the LU based inverse is a little verbose. The 2x2 is okay, but the 3x3 case is quite messy and has six branches.

You can try out inv(SymPy.Sym[Sym("a$i$j") for i = 1:3, j = 1:3])

[KristofferC]

Closing this because a user can easily add their own hand coded inverses if they feel the default speed is not fast enough. Better to provide numerically stable functions by default.