LogAbsDet #3959
LogAbsDet #3959
Conversation
| (x,) = inputs | ||
| (z,) = outputs | ||
| s = numpy.linalg.svd(x, compute_uv=False) | ||
| log_abs_det = numpy.sum(numpy.log(numpy.abs(s))) |
There was a problem hiding this comment.
I didn't check the math. But this can be done by reusing current op. The svd op and the elemwise op. This will be better as the elemwise op will get optimized by Theano and will work on the GPU (for the elemwise).
Then the only think missing to make it work on the GPU would be to have an svn op on the GPU. There is one implementation here that won't request c code: http://scikit-cuda.readthedocs.org/en/latest/generated/skcuda.linalg.svd.html
There was a problem hiding this comment.
In fact, I think it is a good idea to keep this as an op. It will lower memory usage during the gradient on the CPU and will provide a gradient (the currrent svd op don't have a grad implemented).
but an easy way to have this on the GPU would be to make an optimization that will convert this op to a graph that reuse the elemwise on the GPU and make an svd op. For the gradient, MatrixInverse will also need a GPU op to have this work on the GPU. But the first step, is to get this working!
There was a problem hiding this comment.
Yeah, I have no idea what the gradient of SVD would be :) grad of log abs det is very simple on the other hand, so maybe it's better to leave this out of the theano optimizations...
OK didn't know there's no GPU op for MatrixInverse... so step 1 would now be to get LogAbsDet work on GPU? Is it possible to use the skcuda.linalg.svd directly? (I have no idea how the make_thunk works...). So not sure if I should try to do the GPU SVD here directly or in the SVD op? Damn it's still a bit confusing :)
There was a problem hiding this comment.
We didn't document much making new gpu op as we are in transition to a new gpu back-end...
For a first pass, you can make one GPU op that have its gradient on the CPU and use it directly. Then after, you can add an optimization that will convert the CPU op to the GPU op (I can help, it is very easy when you know how to do it).
To make a GPU op, I would use as an example the code of CuFFTOp in sandbox/cuda/fftconv.py. This use scikits.cuda for computation, so you can reuse it for the svd implementation.
There was a problem hiding this comment.
OK thanks, I'll probably get that done some time soon (been a bit busy lately)!
|
Forcing new run of Travis after #3885 has been merged. |
|
This might be a relevant PR for pymc3 too. Any way that the theano simplification would change a manual |
| raise | ||
|
|
||
| def grad(self, inputs, g_outputs): | ||
| gz, = g_outputs |
There was a problem hiding this comment.
The comma here can be hard to spot, why not use single-element list syntax instead?
[gz] = g_outputs
There was a problem hiding this comment.
[gz] = g_outputs
yeah sounds good...
There was a problem hiding this comment.
Any way that the theano simplification would change a manual log(det()) to use this logdet directly?
There may be some problems with that if positivity of det is not enforced... not sure how to handle that...
|
This looks great! Is it possible to automatically optimize |
| s = numpy.linalg.svd(x, compute_uv=False) | ||
| log_abs_det = numpy.sum(numpy.log(numpy.abs(s))) | ||
| z[0] = numpy.asarray(log_abs_det, dtype=x.dtype) | ||
| except Exception: |
There was a problem hiding this comment.
Is it possible to be more specific about the exception type here?
There was a problem hiding this comment.
Is it possible to be more specific about the exception type here?
hmm I guess one should look inside numpy.linalg.svd for similar clauses... or the tests??
Actually I'm pretty swamped with other stuff at the moment, so I'd appreciate if someone else finished this stuff :)
There was a problem hiding this comment.
This looks great! Is it possible to automatically optimize log(det(...)) to logdet(...)?
see here
minor semantic fixes
|
Fixed the tests and did the minor change as suggested by @superbobry but Travis CI seems to be failing... don't know why and I have no time to check now |
|
There was some breakage in the travis setup. A the new run should go fine (assuming there are no problems in this PR). |
|
@nouiz I think this is done now (except for the GPU version) |
|
Now that the release is over, is there anything preventing this from being merged? |
|
I have comments that haven't been done. Mostly, the author started a new jobs, so he have difficulty to find time to finish this. |
|
It should be working now. I fixed the tests too 12 days ago. The only thing missing is the CUDA code, which is of course a major omission... prolly can't find time for that any time soon... |
|
Is it possible to get this merged in without the CUDA code, and seperate that to another PR? |
|
@nouiz What would adding GPU support entail? Writing CUDA? Presumably SVD already exists for GPU. |
Travis CI corrections (Python 3.3 compatibility fixes; PEP8 fixes)
Second go at the LogAbsDet op. Has gradient, but no GPU code so far... if anyone wants to help/ point me to PyCuda/CUDA implementation of SVD, let me know!
This is very similar to numpy.linalg.slogdet, except that the sign of the determinant is omitted, so it just computes log(abs(det(M))) for a matrix M.
Note that M does not need to be positive semidefinite, since we're taking the absolute value (as happens in e.g. maximum likelihood estimation).
do the op for gh-150, the optimization isn't done.