[SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix #5455

MechCoder · 2015-04-10T14:15:19Z

Since sparse matrices now support a isTransposed flag for row major data, DenseMatrices should do the same.

MechCoder · 2015-04-10T14:16:12Z

ping @mengxr Would be great if you could have a look. Also, there is one existing test that fails due to serialization (which I have commented out)

SparkQA · 2015-04-10T15:29:17Z

Test build #30030 has finished for PR 5455 at commit ba02f6e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class FPGrowthModel(JavaModelWrapper):
- class FPGrowth(object):
- class SparseMatrix(Matrix):
- class FlattenedValuesSerializer(BatchedSerializer):
- class ExternalList(object):
- class ExternalListOfList(ExternalList):
- class GroupByKey(object):
- class ExternalGroupBy(ExternalMerger):
This patch does not change any dependencies.

SparkQA · 2015-04-10T18:09:39Z

Test build #30038 has finished for PR 5455 at commit 4e6eaa1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

MechCoder · 2015-04-10T18:14:08Z

The test failure seems unrelated.

SparkQA · 2015-04-13T15:54:27Z

Test build #30168 has finished for PR 5455 at commit 0330540.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

mengxr · 2015-04-13T18:49:10Z

@MechCoder I think you need to update the matrix SerDe in PythonMLlibAPI. Could you double check?

MechCoder · 2015-04-14T17:55:32Z

@mengxr I pushed a non-working update in the last commit just to get feedback.

I went through the code of SerDe in PythonMLlibAPI. I'm unable to understand the entire codepath. I see that somehow, class DenseMatrixPickler is affected but what exactly happens after https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala#L1101 ?
How do I use PickleUtils to store boolean values. Here I converted to an int, and used integer_to_bytes but which exactly is the right way?

SparkQA · 2015-04-14T18:57:16Z

Test build #30262 has finished for PR 5455 at commit 3ba7e25.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

MechCoder · 2015-04-17T06:19:25Z

@mengxr It would be really helpful if you could guide me on my two questions?

SparkQA · 2015-04-17T06:39:21Z

Test build #30457 has finished for PR 5455 at commit 151f3b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

jkbradley · 2015-04-21T06:54:45Z

@MechCoder I'm not an expert here, but I'll ask around. I'm wondering if the failure in PythonMLLibAPISuite indicates that the issue is with the tuple (or tuple code), rather than with how isTransposed is stored.

I'm pretty sure storing the Boolean as an Int is fine.

mengxr · 2015-04-21T07:12:56Z

mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala

@@ -985,6 +985,7 @@ private[spark] object SerDe extends Serializable {
      val m: DenseMatrix = obj.asInstanceOf[DenseMatrix]
      val bytes = new Array[Byte](8 * m.values.size)
      val order = ByteOrder.nativeOrder()
+      val isTransposed = if (m.isTransposed) 1 else 0
      ByteBuffer.wrap(bytes).order(order).asDoubleBuffer().put(m.values)

      out.write(Opcodes.BININT)


Before this line, put out.write(Opcodes.MARK), which marks the start of the tuple.

MechCoder · 2015-04-21T11:25:47Z

@mengxr fixed !

SparkQA · 2015-04-21T13:10:15Z

Test build #30665 has finished for PR 5455 at commit 004a37f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

mengxr · 2015-04-21T17:23:21Z

mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala

      }
      val bytes = getBytes(args(2))
      val n = bytes.length / 8
      val values = new Array[Double](n)
      val order = ByteOrder.nativeOrder()
+      val isTransposed = if (args(3).asInstanceOf[Int] == 1) true else false


Move this line after ByteBuffer.wrap... and simplify it to val isTransposed = args(3).asInstanceOf[Int] == 1.

mengxr · 2015-04-21T17:23:41Z

LGTM except one minor inline comment.

MechCoder · 2015-04-21T17:29:01Z

done

SparkQA · 2015-04-21T19:19:20Z

Test build #30682 has finished for PR 5455 at commit 525c370.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

mengxr · 2015-04-21T21:37:05Z

Merged into master. Thanks!

MechCoder · 2015-04-21T21:44:37Z

@mengxr Thanks! I think we need to support serialization and deserialization for SparseMatrices also?

Since sparse matrices now support a isTransposed flag for row major data, DenseMatrices should do the same. Author: MechCoder <manojkumarsivaraj334@gmail.com> Closes apache#5455 from MechCoder/spark-6845 and squashes the following commits: 525c370 [MechCoder] minor 004a37f [MechCoder] Cast boolean to int 151f3b6 [MechCoder] [WIP] Add isTransposed to pickle DenseMatrix cc0b90a [MechCoder] [SPARK-6845] Add isTranposed flag to DenseMatrix

MechCoder changed the title ~~[SPARK-6845] [Mllib] [PySpark] Add isTranposed flag to DenseMatrix~~ [SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix Apr 10, 2015

MechCoder force-pushed the spark-6845 branch from ba02f6e to 4e6eaa1 Compare April 10, 2015 16:54

MechCoder force-pushed the spark-6845 branch from 4e6eaa1 to 0330540 Compare April 13, 2015 14:41

MechCoder added 2 commits April 17, 2015 11:00

[SPARK-6845] Add isTranposed flag to DenseMatrix

cc0b90a

[WIP] Add isTransposed to pickle DenseMatrix

151f3b6

MechCoder force-pushed the spark-6845 branch from 3ba7e25 to 151f3b6 Compare April 17, 2015 05:31

MechCoder mentioned this pull request Apr 21, 2015

[SPARK-6065] [MLlib] Optimize word2vec.findSynonyms using blas calls #5467

Closed

mengxr reviewed Apr 21, 2015
View reviewed changes

Cast boolean to int

004a37f

mengxr reviewed Apr 21, 2015
View reviewed changes

minor

525c370

asfgit closed this in 45c47fa Apr 21, 2015

MechCoder deleted the spark-6845 branch April 21, 2015 21:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix #5455

[SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix #5455

MechCoder commented Apr 10, 2015

MechCoder commented Apr 10, 2015

SparkQA commented Apr 10, 2015

SparkQA commented Apr 10, 2015

MechCoder commented Apr 10, 2015

SparkQA commented Apr 13, 2015

mengxr commented Apr 13, 2015

MechCoder commented Apr 14, 2015

SparkQA commented Apr 14, 2015

MechCoder commented Apr 17, 2015

SparkQA commented Apr 17, 2015

jkbradley commented Apr 21, 2015

mengxr Apr 21, 2015

MechCoder commented Apr 21, 2015

SparkQA commented Apr 21, 2015

mengxr Apr 21, 2015

mengxr commented Apr 21, 2015

MechCoder commented Apr 21, 2015

SparkQA commented Apr 21, 2015

mengxr commented Apr 21, 2015

MechCoder commented Apr 21, 2015

[SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix #5455

[SPARK-6845] [MLlib] [PySpark] Add isTranposed flag to DenseMatrix #5455

Conversation

MechCoder commented Apr 10, 2015

MechCoder commented Apr 10, 2015

SparkQA commented Apr 10, 2015

SparkQA commented Apr 10, 2015

MechCoder commented Apr 10, 2015

SparkQA commented Apr 13, 2015

mengxr commented Apr 13, 2015

MechCoder commented Apr 14, 2015

SparkQA commented Apr 14, 2015

MechCoder commented Apr 17, 2015

SparkQA commented Apr 17, 2015

jkbradley commented Apr 21, 2015

mengxr Apr 21, 2015

Choose a reason for hiding this comment

MechCoder commented Apr 21, 2015

SparkQA commented Apr 21, 2015

mengxr Apr 21, 2015

Choose a reason for hiding this comment

mengxr commented Apr 21, 2015

MechCoder commented Apr 21, 2015

SparkQA commented Apr 21, 2015

mengxr commented Apr 21, 2015

MechCoder commented Apr 21, 2015