[query] Add method to go from BlockMatrix to NDArray#10398
[query] Add method to go from BlockMatrix to NDArray#10398danking merged 7 commits intohail-is:mainfrom
Conversation
…y. Unclear currently if it should always return 2d things
|
cc @pwc2 , might be of interest to you, though I think you're more interested in going from NDArrays to BlockMatrices. |
|
The tests are going to fail the local backend for now because they use methods like |
| val breezeMat = bm.toBreezeMatrix() | ||
| val shape = IndexedSeq(bm.nRows, bm.nCols) | ||
| // transpose because breeze toArray is column major | ||
| SafeNDArray(shape, breezeMat.t.toArray) |
There was a problem hiding this comment.
If there's no way to avoid an actual transpose, it would be better to do it before collecting to breeze, so you're only transposing the blocks.
There was a problem hiding this comment.
In breeze transpose is just a isTransposed flag, so this should just be telling it which way to collect.
There was a problem hiding this comment.
Right, but then the collect is the transpose. You're collecting a column major matrix in row major order, which for larger matrices will have terrible cache behavior.
There was a problem hiding this comment.
Made the change anyway.
3 participants
I mostly want this for debugging lowering on
LocalBackend, but I added a SparkBackend implementation as well (goes through breeze and java literals, not going to be super performant).