[SPARK-4409][MLlib] Additional Linear Algebra Utils #3319

brkyvz · 2014-11-17T19:07:08Z

Addition of a very limited number of local matrix manipulation and generation methods that would be helpful in the further development for algorithms on top of BlockMatrix (SPARK-3974), such as Randomized SVD, and Multi Model Training (SPARK-1486).
The proposed methods for addition are:

For Matrix

map: maps the values in the matrix with a given function. Produces a new matrix.
update: the values in the matrix are updated with a given function. Occurs in place.

Factory methods for DenseMatrix:

*zeros: Generate a matrix consisting of zeros
*ones: Generate a matrix consisting of ones
*eye: Generate an identity matrix
*rand: Generate a matrix consisting of i.i.d. uniform random numbers
*randn: Generate a matrix consisting of i.i.d. gaussian random numbers
*diag: Generate a diagonal matrix from a supplied vector
*These methods already exist in the factory methods for Matrices, however for cases where we require a DenseMatrix, you constantly have to add .asInstanceOf[DenseMatrix] everywhere, which makes the code "dirtier". I propose moving these functions to factory methods for DenseMatrix where the putput will be a DenseMatrix and the factory methods for Matrices will call these functions directly and output a generic Matrix.

Factory methods for SparseMatrix:

speye: Identity matrix in sparse format. Saves a ton of memory when dimensions are large, especially in Multi Model Training, where each row requires being multiplied by a scalar.
sprand: Generate a sparse matrix with a given density consisting of i.i.d. uniform random numbers.
sprandn: Generate a sparse matrix with a given density consisting of i.i.d. gaussian random numbers.
diag: Generate a diagonal matrix from a supplied vector, but is memory efficient, because it just stores the diagonal. Again, very helpful in Multi Model Training.

Factory methods for Matrices:

Include all the factory methods given above, but return a generic Matrix rather than SparseMatrix or DenseMatrix.
horzCat: Horizontally concatenate matrices to form one larger matrix. Very useful in both Multi Model Training, and for the repartitioning of BlockMatrix.
vertCat: Vertically concatenate matrices to form one larger matrix. Very useful for the repartitioning of BlockMatrix.

The names for these methods were selected from MATLAB

SparkQA · 2014-11-17T19:15:17Z

Test build #23485 has started for PR 3319 at commit 94d7ae9.

This patch merges cleanly.

SparkQA · 2014-11-17T19:16:27Z

Test build #23485 has finished for PR 3319 at commit 94d7ae9.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-17T19:16:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23485/
Test FAILed.

SparkQA · 2014-11-17T19:55:23Z

Test build #23492 has started for PR 3319 at commit d662f9d.

This patch merges cleanly.

SparkQA · 2014-11-17T21:22:21Z

Test build #23492 has finished for PR 3319 at commit d662f9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-17T21:22:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23492/
Test PASSed.

mengxr · 2014-11-25T04:59:43Z

@brkyvz Two comments on the API:

For the APIs we provide, could you add a JAVA test suite and verify that all methods work in Java.
horzCat and vertCat are not MATLAB operators, nor NumPy's. Maybe we should rename them to hstack and vstack, which are at least known by NumPy users.

SparkQA · 2014-11-26T02:10:23Z

Test build #23863 has started for PR 3319 at commit c75f3cd.

This patch merges cleanly.

brkyvz · 2014-11-26T02:12:27Z

@mengxr:
Thanks for the feedback. Added the Java tests!
horzcat and vertcat are in fact MATLAB methods:
http://www.mathworks.com/help/matlab/ref/horzcat.html
http://www.mathworks.com/help/matlab/ref/vertcat.html
They are the underlying methods that are called when someone writes
A = [A1 A2; A3 A4];
I felt the naming was more intuitive as it is like strcat, because you are concatenating matrices either
horizontally or vertically. I'd be happy to change them to hstack and vstack, but horzcat sounds more intuitive to me (maybe I'm biased, because I used to use it more).
Your call :)

SparkQA · 2014-11-26T03:34:09Z

Test build #23863 has finished for PR 3319 at commit c75f3cd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-26T03:34:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23863/
Test PASSed.

SparkQA · 2014-11-26T17:30:14Z

Test build #23900 has started for PR 3319 at commit a8120d2.

This patch merges cleanly.

SparkQA · 2014-11-26T18:55:35Z

Test build #23900 has finished for PR 3319 at commit a8120d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-26T18:55:38Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23900/
Test PASSed.

mengxr · 2014-11-26T20:29:41Z

@brkyvz I didn't know MATLAB has horzcat and vertcat along with [A, B] or [A; B]. I'm okay with adapting method names from MATLAB. Hope there is no copyright issues. (I don't see any special statement from Octave.)

If we want to use MATLAB operators, maybe we should also stick to lowercase method names.

brkyvz · 2014-11-26T21:35:14Z

I checked MATLAB's webpage, I didn't see any copyright mentions for the method names. It's best to triple check though. Since numPy and sciPy share method names with MATLAB, I don't expect there to be problems.
with the last commit I made the method names lowercase :)

mengxr · 2014-12-15T23:22:51Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala

 import breeze.linalg.{Matrix => BM, DenseMatrix => BDM, CSCMatrix => BSM}

+import java.util.{Random, Arrays}
+import scala.collection.mutable.ArrayBuffer


organize imports

mengxr · 2014-12-19T18:33:49Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala

+      }
+      j += 1
+    }
+    while (numCols > lastCol) {


This check is not necessary. At the end of the while (j < numCols) loop, j = numCols + 1. So it is colPtrs(j) = nnz.

SparkQA · 2014-12-20T08:47:30Z

Test build #24668 has started for PR 3319 at commit 10a63a6.

This patch merges cleanly.

SparkQA · 2014-12-20T10:10:21Z

Test build #24668 has finished for PR 3319 at commit 10a63a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-20T10:10:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24668/
Test PASSed.

Some updates for linear algebra utilities

SparkQA · 2014-12-24T07:32:26Z

Test build #24774 has started for PR 3319 at commit 04c4829.

This patch merges cleanly.

SparkQA · 2014-12-24T07:37:46Z

Test build #24775 has started for PR 3319 at commit b0354f6.

This patch merges cleanly.

SparkQA · 2014-12-24T08:36:21Z

Test build #24774 has finished for PR 3319 at commit 04c4829.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T08:36:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24774/
Test FAILed.

SparkQA · 2014-12-24T09:03:37Z

Test build #24775 has finished for PR 3319 at commit b0354f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-24T09:03:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24775/
Test PASSed.

mengxr · 2014-12-29T21:24:41Z

LGTM. Merged into master. Thanks!!

[SPARK-4409] Initial commit to add methods

a14c0da

brkyvz added 2 commits November 17, 2014 11:46

[SPARK-4409] Scalastyle error fixed

83dfe37

[SPARK-4409] Modified according to remote repo

d662f9d

brkyvz force-pushed the SPARK-4409 branch from 94d7ae9 to d662f9d Compare November 17, 2014 19:49

[SPARK-4409] Added JavaAPI Tests, and fixed a couple of bugs

c75f3cd

brkyvz added 2 commits November 26, 2014 09:12

[SPARK-4409] Updated API according to SPARK-4614

f798c82

[SPARK-4409] Finished updates to API according to SPARK-4614

a8120d2

mengxr reviewed Dec 15, 2014
View reviewed changes

mengxr reviewed Dec 19, 2014
View reviewed changes

[SPARK-4409] Fourth pass of code review

10a63a6

mengxr and others added 4 commits December 23, 2014 12:56

simplify fromCOO implementation

4e95e24

update sprand

ecc937a

minor changes

80cfa29

Merge pull request #1 from mengxr/SPARK-4409

04c4829

Some updates for linear algebra utilities

[SPARK-4409] Incorporated mengxr's code

b0354f6

asfgit closed this in 02b55de Dec 29, 2014

brkyvz deleted the SPARK-4409 branch January 30, 2015 20:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-4409][MLlib] Additional Linear Algebra Utils #3319

[SPARK-4409][MLlib] Additional Linear Algebra Utils #3319

brkyvz commented Nov 17, 2014

SparkQA commented Nov 17, 2014

SparkQA commented Nov 17, 2014

AmplabJenkins commented Nov 17, 2014

SparkQA commented Nov 17, 2014

SparkQA commented Nov 17, 2014

AmplabJenkins commented Nov 17, 2014

mengxr commented Nov 25, 2014

SparkQA commented Nov 26, 2014

brkyvz commented Nov 26, 2014

SparkQA commented Nov 26, 2014

AmplabJenkins commented Nov 26, 2014

SparkQA commented Nov 26, 2014

SparkQA commented Nov 26, 2014

AmplabJenkins commented Nov 26, 2014

mengxr commented Nov 26, 2014

brkyvz commented Nov 26, 2014

mengxr Dec 15, 2014

mengxr Dec 19, 2014

SparkQA commented Dec 20, 2014

SparkQA commented Dec 20, 2014

AmplabJenkins commented Dec 20, 2014

SparkQA commented Dec 24, 2014

SparkQA commented Dec 24, 2014

SparkQA commented Dec 24, 2014

AmplabJenkins commented Dec 24, 2014

SparkQA commented Dec 24, 2014

AmplabJenkins commented Dec 24, 2014

mengxr commented Dec 29, 2014

[SPARK-4409][MLlib] Additional Linear Algebra Utils #3319

[SPARK-4409][MLlib] Additional Linear Algebra Utils #3319

Conversation

brkyvz commented Nov 17, 2014

SparkQA commented Nov 17, 2014

SparkQA commented Nov 17, 2014

AmplabJenkins commented Nov 17, 2014

SparkQA commented Nov 17, 2014

SparkQA commented Nov 17, 2014

AmplabJenkins commented Nov 17, 2014

mengxr commented Nov 25, 2014

SparkQA commented Nov 26, 2014

brkyvz commented Nov 26, 2014

SparkQA commented Nov 26, 2014

AmplabJenkins commented Nov 26, 2014

SparkQA commented Nov 26, 2014

SparkQA commented Nov 26, 2014

AmplabJenkins commented Nov 26, 2014

mengxr commented Nov 26, 2014

brkyvz commented Nov 26, 2014

mengxr Dec 15, 2014

Choose a reason for hiding this comment

mengxr Dec 19, 2014

Choose a reason for hiding this comment

SparkQA commented Dec 20, 2014

SparkQA commented Dec 20, 2014

AmplabJenkins commented Dec 20, 2014

SparkQA commented Dec 24, 2014

SparkQA commented Dec 24, 2014

SparkQA commented Dec 24, 2014

AmplabJenkins commented Dec 24, 2014

SparkQA commented Dec 24, 2014

AmplabJenkins commented Dec 24, 2014

mengxr commented Dec 29, 2014