[SPARK-7681][MLlib] Add SparseVector support for gemv #6209

viirya · 2015-05-16T05:07:19Z

JIRA: https://issues.apache.org/jira/browse/SPARK-7681

SparkQA · 2015-05-16T05:23:19Z

Test build #32885 has finished for PR 6209 at commit c069507.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2015-05-16T05:29:22Z

retest this please.

SparkQA · 2015-05-16T07:16:37Z

Test build #32889 has finished for PR 6209 at commit c069507.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dbtsai · 2015-05-17T04:44:27Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala

+        case (dense: DenseMatrix, dx: DenseVector) =>
+          gemv(alpha, dense, dx, beta, y)
+        case (dense: DenseMatrix, sx: SparseVector) =>
+          gemv(alpha, dense, sx, beta, y)
        case _ =>


How about SparseMatrix and SparseVector? To make the consistent naming, we can use dmA, smA, dvx, and svx.

If you don't really want to add SparseMatrix and SparseVector, the type safety will be broken when you call with this configuration. Previously, this function is totally type safe in compile time, and no way to get into "case _".

SparkQA · 2015-05-17T15:53:20Z

Test build #32936 has finished for PR 6209 at commit 4616696.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2015-05-17T16:05:58Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala

+    var xNnz = xIndices.size
+    var xValues = x.values
+
+    scal(beta, y)


Why we should check it?

dbtsai · 2015-05-17T19:20:23Z

Finally in sealed trait Matrix, change

  /** Convenience method for `Matrix`-`DenseVector` multiplication. */
  def multiply(y: DenseVector): DenseVector = {

to

  /** Convenience method for `Matrix`-`Vector` multiplication. */
  def multiply(y: Vector): DenseVector = {

debasish83 · 2015-05-17T19:34:03Z

Are there runtime comparisons posted with vector*vector operations for these changes BLAS-1 vs BLAS-2 ? SparseMatrix * SparseVector compared to Array[SparseVector] x SparseVector

dbtsai · 2015-05-17T20:18:53Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala

+          var i = Acols(colCounterForA)
+          val indEnd = Acols(colCounterForA + 1)
+
+          val xVal = xValues(k) * alpha


xVal will be easily to read as xValues. simply make it as xTemp

dbtsai · 2015-05-17T20:19:24Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala

+          val xVal = xValues(k) * alpha
+          while (i < indEnd) {
+            val rowIndex = Arows(i)
+            yValues(rowIndex) += Avals(i) * xVal


yValues(Arows(i)) += Avals(i) * xTemp

SparkQA · 2015-05-18T07:55:53Z

Test build #32981 has finished for PR 6209 at commit 410381a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-18T08:14:36Z

Test build #32982 has finished for PR 6209 at commit 054f05d.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

viirya · 2015-05-18T08:33:32Z

retest this please.

SparkQA · 2015-05-18T08:51:28Z

Test build #32986 has finished for PR 6209 at commit 054f05d.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-18T09:10:53Z

Test build #32988 has finished for PR 6209 at commit 458d1ae.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-05-18T11:12:41Z

Test build #32989 has finished for PR 6209 at commit 57a8c1e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dbtsai · 2015-05-18T15:57:22Z

LGTM cc: @mengxr @srowen @jkbradley

mengxr · 2015-05-18T19:16:12Z

I will take a pass on the changes.

mengxr · 2015-05-18T23:27:25Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala

-  /** Convenience method for `Matrix`-`DenseVector` multiplication. */
-  def multiply(y: DenseVector): DenseVector = {
+  /** Convenience method for `Matrix`-`Vector` multiplication. */
+  def multiply(y: Vector): DenseVector = {


We cannot delete def multiply(y: DenseVector), which breaks binary compatibility. Please delegate the implementation to multiply(y: Vector) and update Mima excludes.

viirya · 2015-05-19T01:23:44Z

I just found the original gemv for SparseMatrix and DenseVector has a bug that it only scales y when beta != 0.0. We should fix it too.

SparkQA · 2015-05-19T03:05:53Z

Test build #33040 has finished for PR 6209 at commit b890e63.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaOneVsRestExample
- head("OneVsRest Example: multiclass to binary reduction using OneVsRest")
- class DataTypeSingleton(type):
- class NullType(DataType):
- class AtomicType(DataType):
- class NumericType(AtomicType):
- class IntegralType(NumericType):
- class FractionalType(NumericType):
- class StringType(AtomicType):
- class BinaryType(AtomicType):
- class BooleanType(AtomicType):
- class DateType(AtomicType):
- class TimestampType(AtomicType):
- class DecimalType(FractionalType):
- class DoubleType(FractionalType):
- class FloatType(FractionalType):
- class ByteType(IntegralType):
- class IntegerType(IntegralType):
- class LongType(IntegralType):
- class ShortType(IntegralType):
- class Column(object):
- class GroupedData(object):

SparkQA · 2015-05-19T03:14:43Z

Test build #33042 has finished for PR 6209 at commit ce0bb8b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

JIRA: https://issues.apache.org/jira/browse/SPARK-7681 Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #6209 from viirya/sparsevector_gemv and squashes the following commits: ce0bb8b [Liang-Chi Hsieh] Still need to scal y when beta is 0.0 because it clears out y. b890e63 [Liang-Chi Hsieh] Do not delete multiply for DenseVector. 57a8c1e [Liang-Chi Hsieh] Add MimaExcludes for v1.4. 458d1ae [Liang-Chi Hsieh] List DenseMatrix.multiply and SparseMatrix.multiply to MimaExcludes too. 054f05d [Liang-Chi Hsieh] Fix scala style. 410381a [Liang-Chi Hsieh] Address comments. Make Matrix.multiply more generalized. 4616696 [Liang-Chi Hsieh] Add support for SparseVector with SparseMatrix. 5d6d07a [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into sparsevector_gemv c069507 [Liang-Chi Hsieh] Add SparseVector support for gemv with DenseMatrix. (cherry picked from commit d03638c) Signed-off-by: Xiangrui Meng <meng@databricks.com>

mengxr · 2015-05-19T04:39:10Z

LGTM. Merged into master and branch-1.4. I will test the Mima excludes to make them minimal. Please update the naive Bayes PR. Thanks!

JIRA: https://issues.apache.org/jira/browse/SPARK-7681 Author: Liang-Chi Hsieh <viirya@gmail.com> Closes apache#6209 from viirya/sparsevector_gemv and squashes the following commits: ce0bb8b [Liang-Chi Hsieh] Still need to scal y when beta is 0.0 because it clears out y. b890e63 [Liang-Chi Hsieh] Do not delete multiply for DenseVector. 57a8c1e [Liang-Chi Hsieh] Add MimaExcludes for v1.4. 458d1ae [Liang-Chi Hsieh] List DenseMatrix.multiply and SparseMatrix.multiply to MimaExcludes too. 054f05d [Liang-Chi Hsieh] Fix scala style. 410381a [Liang-Chi Hsieh] Address comments. Make Matrix.multiply more generalized. 4616696 [Liang-Chi Hsieh] Add support for SparseVector with SparseMatrix. 5d6d07a [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into sparsevector_gemv c069507 [Liang-Chi Hsieh] Add SparseVector support for gemv with DenseMatrix.

Add SparseVector support for gemv with DenseMatrix.

c069507

viirya mentioned this pull request May 16, 2015

[SPARK-7652][MLlib] Update the implementation of naive Bayes prediction with BLAS #6189

Closed

dbtsai reviewed May 17, 2015
View reviewed changes

viirya added 2 commits May 17, 2015 21:47

Merge remote-tracking branch 'upstream/master' into sparsevector_gemv

5d6d07a

Add support for SparseVector with SparseMatrix.

4616696

viirya changed the title ~~[SPARK-7681][MLlib] Add SparseVector support for gemv with DenseMatrix~~ [SPARK-7681][MLlib] Add SparseVector support for gemv May 17, 2015

viirya reviewed May 17, 2015
View reviewed changes

dbtsai reviewed May 17, 2015
View reviewed changes

Address comments. Make Matrix.multiply more generalized.

410381a

Fix scala style.

054f05d

List DenseMatrix.multiply and SparseMatrix.multiply to MimaExcludes too.

458d1ae

Add MimaExcludes for v1.4.

57a8c1e

mengxr reviewed May 18, 2015
View reviewed changes

Do not delete multiply for DenseVector.

b890e63

Still need to scal y when beta is 0.0 because it clears out y.

ce0bb8b

asfgit closed this in d03638c May 19, 2015

debasish83 mentioned this pull request May 19, 2015

[WIP][MLLIB][SPARK-4675][SPARK-4823]RowSimilarity #6213

Closed

viirya deleted the sparsevector_gemv branch December 27, 2023 18:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7681][MLlib] Add SparseVector support for gemv #6209

[SPARK-7681][MLlib] Add SparseVector support for gemv #6209

viirya commented May 16, 2015

SparkQA commented May 16, 2015

viirya commented May 16, 2015

SparkQA commented May 16, 2015

dbtsai May 17, 2015

dbtsai May 17, 2015

SparkQA commented May 17, 2015

viirya May 17, 2015

dbtsai commented May 17, 2015

debasish83 commented May 17, 2015

dbtsai May 17, 2015

dbtsai May 17, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

viirya commented May 18, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

dbtsai commented May 18, 2015

mengxr commented May 18, 2015

mengxr May 18, 2015

viirya commented May 19, 2015

SparkQA commented May 19, 2015

SparkQA commented May 19, 2015

mengxr commented May 19, 2015

[SPARK-7681][MLlib] Add SparseVector support for gemv #6209

[SPARK-7681][MLlib] Add SparseVector support for gemv #6209

Conversation

viirya commented May 16, 2015

SparkQA commented May 16, 2015

viirya commented May 16, 2015

SparkQA commented May 16, 2015

dbtsai May 17, 2015

Choose a reason for hiding this comment

dbtsai May 17, 2015

Choose a reason for hiding this comment

SparkQA commented May 17, 2015

viirya May 17, 2015

Choose a reason for hiding this comment

dbtsai commented May 17, 2015

debasish83 commented May 17, 2015

dbtsai May 17, 2015

Choose a reason for hiding this comment

dbtsai May 17, 2015

Choose a reason for hiding this comment

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

viirya commented May 18, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

SparkQA commented May 18, 2015

dbtsai commented May 18, 2015

mengxr commented May 18, 2015

mengxr May 18, 2015

Choose a reason for hiding this comment

viirya commented May 19, 2015

SparkQA commented May 19, 2015

SparkQA commented May 19, 2015

mengxr commented May 19, 2015