Randomized SVD

[hunse]

Added a randomized SVD solver. This is pretty fast for larger matrices (on my system, it gets faster than cholesky for around 650 neurons, and can solve a system with 60000 eval points and 10000 neurons in about 6 seconds).

[tbekolay]

FYI I'm trying to fix the TravisCI errors now, so don't worry about those.

[hunse]

Doesn't it just need a rebase?

[tbekolay]

No, recently Travis builds have been failing for conda-related reasons.

[Seanny123]

Overall this commit looks good to me, but it should be noted that although I familiar with the idea of decoder solving, I'm not clear on the algorithm for randomized SVD. Eric, would you like to go over the algorithm with on Monday so I can truly say I understand this pull request?

[hunse]

The algorithm isn't in our repo, I just call out to sklearn.

[Seanny123]

Sorry, I meant the stuff you do from line 230 and onwards.

[hunse]

That's the same as for the basic SVD solver. See if you can prove it for yourself, starting from the normal equation (A^T A + sigma^2 I) x = A^T y (this is an equivalent linear system to the least squares problem we want to solve) and using the fact that A = U S V, where U and V are orthogonal and S is diagonal.

[tbekolay]

Rebased this and squashed the history. I used this a while ago, and it worked as advertised! Will merge tomorrow morning if there are no objections.

[jgosmann]

How do I use this solver? If I do model.config[nengo.Connection].solver = nengo.solvers.randomized_svd, I get ValueError: '<function randomized_svd at 0x7f0c2ab64758>' is not a solver. It seems that a corresponding class deriving from Solver is missing?

[jgosmann]

I think, I figured it out. It can be passed as argument to LstsqL2 class.

[hunse]

Yep, exactly, you've got it. That should probably be documented better somewhere.