-
Notifications
You must be signed in to change notification settings - Fork 46
Linear Algebra Cheat Sheet
Compared to other numerical computing environments, Marlin matrices default to column major ordering, like Breeze, but indexing is 0-based, like Numpy.
sc represents created SparkContxt, rows represents the number of rows of the matrix, columns represents the number of columns of the matrix, path represents the path in hdfs.
| Operation | API |
| Zeroed matrix | MTUtils.zerosDenVecMatrix(sc, rows, columns) |
| Ones matrix | MTUtils.onesDenVecMatrix(sc, rows, columns) |
| Random elements (from 0 to 1) matrix | MTUtils.randomDenVecMatrix(sc, rows, columns) |
| Matrix creation from two-dimension array | MTUtils.arrayToMatrix(sc, array, partitions) |
| DenseVecMatrix creation from text file | MTUtils.loadMatrixFile(sc, path) |
| DenseVecMatrix creation from text files in a directory | MTUtils.loadMatrixFiles(sc, path) |
| Blockmatrix creation from text file | MTUtils.loadBlockMatrixFile(sc, path) |
| DenseVecMatrix creation from sequence file | MTUtils.loadMatrixSeqFile(sc, path) |
API randomDenVecMatrix default gengerates elements from 0 to 1 in uniform distribution, if you want to gengerate elemnts value with different range, you can use code below
import import edu.nju.pasalab.marlin.utils._
val generator = new UniformGenerator(-1.0, 1.0)
val rows, cols = 10000
val mat = MTUtils.randomDenVecMatrix(sc, rows, cols, generator)Except uniform distribution, we also support PoissonGenerator(mean: Double) and StandardNormalGenerator(mean: Double, variance: Double).
When load matrix from files, the orginal format can be as like 2:1, 0.0, 4 9.1, then call method MTUtils.loadMatrixFile, it will generate a pair (2L, DenseVector(1.0,0.0,4.0,9.1)) in RDD.
Currently, these APIs are only suporrted in DenseVecMatrix type, parameters startRow, endRow, startCol and endCol are all included boundaries.
| Operation | API |
| Extract rows of matrix | sliceByRow(startRow, endRow) |
| Extract columns of matrix | sliceByColumn(startCol, endCol) |
| get sub-Matrix | getSubMatrix(startRow, endRow, startCol, endCol) |
| Operation | API |
| Matrix-matrix addition | add(other: DistributedMatrix) |
| Matrix-matrix subtraction | subtract(other: DistributedMatrix)/td> |
| Matrix-matrix multiplication | multiply(other: DistributedMatrix, cores: Int) |
| Element-wise addition | add(b: Double) |
| Element-wise subtraction | subtract(b: Double) / substactBy(b: Double) |
| Element-wise multiplication | multiply(b: Double) |
| Element-wise division | divide(b: Double) / divideBy(b: Double) |
| transpose of matrix | transpose() |
| lu decompose of matrix | luDecompose() |
| cholesky decompose of matrix | choleskyDecompose() |
| inverse of matrix | inverse() |
| Operation | API |
| Save the matrix to hdfs in text format | saveToFileSystem(path) |
| Save the matrix to hdfs in SequenceFile format | saveSequenceFile(path) |
If the original matrix is in DenseVecMatrix type, then call saveToFileSystem(path) method will save the matrix in the DenseVecMatrix text format. The output directory will store these files in which each line may be unordered. You can then call MTUtils.loadMatrixFiles(sc, path) method to create matrix from these files.
On the other hand, if the original matrix is in BlockMatrix type, call saveToFileSystem(path, "blockmatrix") method will save the matrix in the BlockMatrix text format. Otherwise, you can call saveToFileSystem(path), it will do another transformation from BlockMatrix to DenseVecMatrix and store the result in DenseVecMatrix text format.
| print the matrix out to screen, easy usage for developers | print() |