[SPARK-5990] [MLLIB] Model import/export for IsotonicRegression #5270

yanboliang · 2015-03-30T16:25:00Z

Model import/export for IsotonicRegression

SparkQA · 2015-03-30T16:28:18Z

Test build #29410 has started for PR 5270 at commit 2b2f5a1.

SparkQA · 2015-03-30T17:54:07Z

Test build #29410 has finished for PR 5270 at commit 2b2f5a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Data(boundaries: Array[Double], predictions: Array[Double], isotonic: Boolean)
This patch does not change any dependencies.

AmplabJenkins · 2015-03-30T17:54:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29410/
Test PASSed.

mengxr · 2015-03-30T23:55:25Z

mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala

+    def thisClassName: String = "org.apache.spark.mllib.regression.IsotonicRegressionModel"
+
+    /** Model data for model import/export */
+    case class Data(boundaries: Array[Double], predictions: Array[Double], isotonic: Boolean)


It would be easier to inspect the data file if we put each interval as a record. For example:

boundary prediction

0.0 -1.0

1.0 0.5

2.0 1.0

We can save isotonic as a value in the metadata.

AmplabJenkins · 2015-04-04T18:56:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29715/
Test FAILed.

mengxr · 2015-04-05T23:06:02Z

mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala

+    def thisClassName: String = "org.apache.spark.mllib.regression.IsotonicRegressionModel"
+
+    /** Model data for model import/export */
+    case class Data(intervals: Array[(Double, Double)])


My suggestion was

case class Data(boundary: Double, prediction: Double)

And then save each (boundary, prediction) pair as a record:

sqlContext.createDataFrame(boundaries.zip(predictions).map { case (b, p) => Data(b, p) }) .saveAsParquetFile(dataPath(path))

SparkQA · 2015-04-09T18:03:23Z

Test build #29953 has started for PR 5270 at commit 49600cc.

SparkQA · 2015-04-09T19:24:07Z

Test build #29953 has finished for PR 5270 at commit 49600cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Data(boundary: Double, prediction: Double)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-09T19:24:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29953/
Test FAILed.

mengxr · 2015-04-09T20:27:41Z

test this please

SparkQA · 2015-04-09T20:33:29Z

Test build #29960 has started for PR 5270 at commit 49600cc.

SparkQA · 2015-04-09T21:32:14Z

Test build #29960 has finished for PR 5270 at commit 49600cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Data(boundary: Double, prediction: Double)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-09T21:32:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29960/
Test FAILed.

mengxr · 2015-04-09T22:21:10Z

mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala

+
+  import org.apache.spark.mllib.util.Loader._
+
+  private  object SaveLoadV1_0 {


remove one space after private

SparkQA · 2015-04-20T16:48:32Z

Test build #30593 has started for PR 5270 at commit f80ec1b.

SparkQA · 2015-04-20T18:30:46Z

Test build #30593 has finished for PR 5270 at commit f80ec1b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Data(boundary: Double, prediction: Double)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-20T18:30:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30593/
Test PASSed.

mengxr · 2015-04-20T18:44:20Z

mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala

+        predictions: Array[Double], 
+        isotonic: Boolean): Unit = {
+      val sqlContext = new SQLContext(sc)
+      import sqlContext.implicits._


Remove this line because no implicits are used.

mengxr · 2015-04-20T18:50:47Z

LGTM except minor inline comments.

mengxr · 2015-04-20T18:51:16Z

mllib/src/test/scala/org/apache/spark/mllib/regression/IsotonicRegressionSuite.scala

+      val sameModel = IsotonicRegressionModel.load(sc, path)
+      assert(model.boundaries === sameModel.boundaries)
+      assert(model.predictions === sameModel.predictions)
+      assert(model.isotonic == model.isotonic)


== -> ===

SparkQA · 2015-04-21T05:13:31Z

Test build #30635 has started for PR 5270 at commit 872028d.

SparkQA · 2015-04-21T06:55:38Z

Test build #30635 has finished for PR 5270 at commit 872028d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Data(boundary: Double, prediction: Double)
This patch does not change any dependencies.

AmplabJenkins · 2015-04-21T06:55:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30635/
Test PASSed.

mengxr · 2015-04-21T07:14:35Z

Merged into master. Thanks!

Model import/export for IsotonicRegression Author: Yanbo Liang <ybliang8@gmail.com> Closes apache#5270 from yanboliang/spark-5990 and squashes the following commits: 872028d [Yanbo Liang] fix code style f80ec1b [Yanbo Liang] address comments 49600cc [Yanbo Liang] address comments 429ff7d [Yanbo Liang] store each interval as a record 2b2f5a1 [Yanbo Liang] Model import/export for IsotonicRegression

Model import/export for IsotonicRegression

2b2f5a1

mengxr reviewed Mar 30, 2015
View reviewed changes

store each interval as a record

429ff7d

mengxr reviewed Apr 5, 2015
View reviewed changes

address comments

49600cc

mengxr reviewed Apr 9, 2015
View reviewed changes

address comments

f80ec1b

mengxr reviewed Apr 20, 2015
View reviewed changes

fix code style

872028d

asfgit closed this in 1f2f723 Apr 21, 2015

yanboliang deleted the spark-5990 branch April 24, 2015 10:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-5990] [MLLIB] Model import/export for IsotonicRegression #5270

[SPARK-5990] [MLLIB] Model import/export for IsotonicRegression #5270

yanboliang commented Mar 30, 2015

SparkQA commented Mar 30, 2015

SparkQA commented Mar 30, 2015

AmplabJenkins commented Mar 30, 2015

mengxr Mar 30, 2015

AmplabJenkins commented Apr 4, 2015

mengxr Apr 5, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 9, 2015

AmplabJenkins commented Apr 9, 2015

mengxr commented Apr 9, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 9, 2015

AmplabJenkins commented Apr 9, 2015

mengxr Apr 9, 2015

SparkQA commented Apr 20, 2015

SparkQA commented Apr 20, 2015

AmplabJenkins commented Apr 20, 2015

mengxr Apr 20, 2015

mengxr commented Apr 20, 2015

mengxr Apr 20, 2015

SparkQA commented Apr 21, 2015

SparkQA commented Apr 21, 2015

AmplabJenkins commented Apr 21, 2015

mengxr commented Apr 21, 2015


		import org.apache.spark.mllib.util.Loader._

		private object SaveLoadV1_0 {

[SPARK-5990] [MLLIB] Model import/export for IsotonicRegression #5270

[SPARK-5990] [MLLIB] Model import/export for IsotonicRegression #5270

Conversation

yanboliang commented Mar 30, 2015

SparkQA commented Mar 30, 2015

SparkQA commented Mar 30, 2015

AmplabJenkins commented Mar 30, 2015

mengxr Mar 30, 2015

Choose a reason for hiding this comment

AmplabJenkins commented Apr 4, 2015

mengxr Apr 5, 2015

Choose a reason for hiding this comment

SparkQA commented Apr 9, 2015

SparkQA commented Apr 9, 2015

AmplabJenkins commented Apr 9, 2015

mengxr commented Apr 9, 2015

SparkQA commented Apr 9, 2015

SparkQA commented Apr 9, 2015

AmplabJenkins commented Apr 9, 2015

mengxr Apr 9, 2015

Choose a reason for hiding this comment

SparkQA commented Apr 20, 2015

SparkQA commented Apr 20, 2015

AmplabJenkins commented Apr 20, 2015

mengxr Apr 20, 2015

Choose a reason for hiding this comment

mengxr commented Apr 20, 2015

mengxr Apr 20, 2015

Choose a reason for hiding this comment

SparkQA commented Apr 21, 2015

SparkQA commented Apr 21, 2015

AmplabJenkins commented Apr 21, 2015

mengxr commented Apr 21, 2015