[SPARK-8711] [ML] Add additional methods to PySpark ML tree models #7095

MechCoder · 2015-06-29T19:45:49Z

Add numNodes and depth to treeModels, add treeWeights to ensemble Models.
Add repr to all models.

AmplabJenkins · 2015-06-29T19:47:14Z

Merged build triggered.

AmplabJenkins · 2015-06-29T19:47:20Z

Merged build started.

MechCoder · 2015-06-29T19:48:38Z

@mengxr Should we add wrapper around the trees methos in ensembles as well? I'm not sure how we would manipulate java objects in python.

AmplabJenkins · 2015-06-29T19:52:12Z

Merged build triggered.

AmplabJenkins · 2015-06-29T19:52:20Z

Merged build started.

SparkQA · 2015-06-29T19:55:19Z

Test build #36023 has started for PR 7095 at commit 849aabe.

AmplabJenkins · 2015-06-29T20:02:32Z

Merged build finished. Test FAILed.

SparkQA · 2015-06-29T20:06:46Z

Test build #36023 has finished for PR 7095 at commit 849aabe.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(treeEnsembleModels):
- class GBTClassificationModel(treeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class treeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(treeEnsembleModels):
- class GBTRegressionModel(treeEnsembleModels):

AmplabJenkins · 2015-06-29T20:07:09Z

Merged build finished. Test PASSed.

feynmanliang · 2015-06-29T23:47:06Z

python/pyspark/ml/regression.py

@@ -409,6 +444,8 @@ class GBTRegressor(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,
    ...     (0.0, Vectors.sparse(1, [], []))], ["label", "features"])
    >>> gbt = GBTRegressor(maxIter=5, maxDepth=2)
    >>> model = gbt.fit(df)
+    >>> model.treeWeights
+    [1.0, 0.1, 0.1, 0.1, 0.1]


Floating point equality is causing this test to fail. Try numpy.allclose

MechCoder · 2015-06-30T04:59:33Z

@feynmanliang Thanks. I have addressed your comments.

AmplabJenkins · 2015-06-30T05:02:11Z

Merged build triggered.

AmplabJenkins · 2015-06-30T05:02:17Z

Merged build started.

SparkQA · 2015-06-30T05:05:31Z

Test build #36090 has started for PR 7095 at commit 2fde31a.

SparkQA · 2015-06-30T05:16:27Z

Test build #36090 has finished for PR 7095 at commit 2fde31a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-06-30T05:16:50Z

Merged build finished. Test PASSed.

JoshRosen · 2015-06-30T05:19:43Z

Uh oh: looks like this is actually failing some tests:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "./python/run-tests.py", line 166, in process_queue
    run_individual_python_test(test_goal, python_exec)
  File "./python/run-tests.py", line 74, in run_individual_python_test
    if not re.match('[0-9]+', line):
  File "/home/anaconda/envs/py3k/lib/python3.4/re.py", line 160, in match
    return _compile(pattern, flags).match(string)
TypeError: can't use a string pattern on a bytes-like object

Finished test(python3.4): pyspark.ml.recommendation (15s)
Finished test(python3.4): pyspark.ml.feature (18s)
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/home/anaconda/envs/py3k/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "./python/run-tests.py", line 166, in process_queue
    run_individual_python_test(test_goal, python_exec)
  File "./python/run-tests.py", line 74, in run_individual_python_test
    if not re.match('[0-9]+', line):
  File "/home/anaconda/envs/py3k/lib/python3.4/re.py", line 160, in match
    return _compile(pattern, flags).match(string)
TypeError: can't use a string pattern on a bytes-like object

This is the consequence of not having good test coverage on the test script... I'll fix this shortly.

JoshRosen · 2015-06-30T05:41:22Z

I've opened #7112 to hotfix the build issue which masked the test failure here.

JoshRosen · 2015-06-30T06:09:44Z

Jenkins, retest this please.

AmplabJenkins · 2015-06-30T06:12:12Z

Merged build triggered.

AmplabJenkins · 2015-06-30T06:12:18Z

Merged build started.

SparkQA · 2015-06-30T06:16:00Z

Test build #36096 has started for PR 7095 at commit 2fde31a.

SparkQA · 2015-06-30T06:26:44Z

Test build #36096 has finished for PR 7095 at commit 2fde31a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-06-30T06:26:50Z

Merged build finished. Test FAILed.

JoshRosen · 2015-06-30T06:29:14Z

Now that the test reporting is fixed, we can see the failure:

**********************************************************************
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/regression.py", line 175, in __main__.DecisionTreeRegressor
Failed example:
    model.depth
Expected:
    2
Got:
    1
**********************************************************************
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/ml/regression.py", line 177, in __main__.DecisionTreeRegressor
Failed example:
    model.numNodes
Expected:
    1
Got:
    3
**********************************************************************
   2 of  10 in __main__.DecisionTreeRegressor
***Test Failed*** 2 failures.

JoshRosen · 2015-06-30T06:30:18Z

By the way, check out --help on python/run-tests if you haven't already; there's some neat options for controlling which suites, python versions, and degree of parallelism to use, which might be a big productivity boost for you given all of the Python work that you've been doing lately.

AmplabJenkins · 2015-06-30T12:52:11Z

Merged build triggered.

MechCoder · 2015-07-06T19:41:48Z

I renamed pyTreeWeights to javaTreeWeights.

AmplabJenkins · 2015-07-06T19:43:11Z

Merged build triggered.

AmplabJenkins · 2015-07-06T19:43:21Z

Merged build started.

SparkQA · 2015-07-06T19:44:37Z

Test build #36598 has started for PR 7095 at commit 38a0860.

SparkQA · 2015-07-06T20:19:00Z

Test build #36598 has finished for PR 7095 at commit 38a0860.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-07-06T20:19:27Z

Merged build finished. Test PASSed.

mengxr · 2015-07-06T23:24:31Z

mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala

@@ -70,6 +71,10 @@ private[ml] trait TreeEnsembleModel {
  /** Weights for each tree, zippable with [[trees]] */
  def treeWeights: Array[Double]

+  /** Weights used by the python wrappers. */
+  // Note: An array cannot be returned directly due to serialization problems.
+  def javaTreeWeights: Vector = Vectors.dense(treeWeights)


Make it package private. Java users do not really need it.

mengxr · 2015-07-06T23:25:15Z

LGTM except that the utility method should be package private.

MechCoder · 2015-07-07T04:49:44Z

done. I could not make it private[python] . It was giving some error.

AmplabJenkins · 2015-07-07T04:53:11Z

Merged build triggered.

AmplabJenkins · 2015-07-07T04:53:19Z

Merged build started.

AmplabJenkins · 2015-07-07T05:08:20Z

Merged build finished. Test FAILed.

MechCoder · 2015-07-07T06:48:03Z

jenkins retest this please

AmplabJenkins · 2015-07-07T06:48:12Z

Merged build triggered.

AmplabJenkins · 2015-07-07T06:48:21Z

Merged build started.

SparkQA · 2015-07-07T06:49:32Z

Test build #36652 has started for PR 7095 at commit 23b08be.

SparkQA · 2015-07-07T06:59:35Z

Test build #36652 has finished for PR 7095 at commit 23b08be.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-07-07T06:59:41Z

Merged build finished. Test FAILed.

MechCoder · 2015-07-07T13:29:21Z

jenkins retest this please

mengxr · 2015-07-07T15:05:27Z

test this please

AmplabJenkins · 2015-07-07T15:08:12Z

Merged build triggered.

AmplabJenkins · 2015-07-07T15:08:18Z

Merged build started.

SparkQA · 2015-07-07T15:08:26Z

Test build #36690 has started for PR 7095 at commit 23b08be.

SparkQA · 2015-07-07T15:42:24Z

Test build #36690 has finished for PR 7095 at commit 23b08be.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):

AmplabJenkins · 2015-07-07T15:42:54Z

Merged build finished. Test PASSed.

mengxr · 2015-07-07T15:58:57Z

Merged into master. Thanks!

MechCoder force-pushed the missing_methods_tree branch from ce343b9 to 849aabe Compare June 29, 2015 19:47

feynmanliang reviewed Jun 29, 2015
View reviewed changes

MechCoder force-pushed the missing_methods_tree branch from 2fde31a to a6a27ca Compare June 30, 2015 12:52

rename pyTreeWeights to javaTreeWeights

38a0860

mengxr reviewed Jul 6, 2015
View reviewed changes

private [spark]

23b08be

asfgit closed this in 1dbc4a1 Jul 7, 2015

MechCoder deleted the missing_methods_tree branch July 7, 2015 16:23

[SPARK-8711] [ML] Add additional methods to PySpark ML tree models #7095

[SPARK-8711] [ML] Add additional methods to PySpark ML tree models #7095

Conversation

MechCoder commented Jun 29, 2015

AmplabJenkins commented Jun 29, 2015

AmplabJenkins commented Jun 29, 2015

MechCoder commented Jun 29, 2015

AmplabJenkins commented Jun 29, 2015

AmplabJenkins commented Jun 29, 2015

SparkQA commented Jun 29, 2015

AmplabJenkins commented Jun 29, 2015

SparkQA commented Jun 29, 2015

AmplabJenkins commented Jun 29, 2015

feynmanliang Jun 29, 2015

Choose a reason for hiding this comment

MechCoder commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

JoshRosen commented Jun 30, 2015

JoshRosen commented Jun 30, 2015

JoshRosen commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

SparkQA commented Jun 30, 2015

SparkQA commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

JoshRosen commented Jun 30, 2015

JoshRosen commented Jun 30, 2015

AmplabJenkins commented Jun 30, 2015

MechCoder commented Jul 6, 2015

AmplabJenkins commented Jul 6, 2015

AmplabJenkins commented Jul 6, 2015

SparkQA commented Jul 6, 2015

SparkQA commented Jul 6, 2015

AmplabJenkins commented Jul 6, 2015

mengxr Jul 6, 2015

Choose a reason for hiding this comment

mengxr commented Jul 6, 2015

MechCoder commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

MechCoder commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

SparkQA commented Jul 7, 2015

SparkQA commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

MechCoder commented Jul 7, 2015

mengxr commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

SparkQA commented Jul 7, 2015

SparkQA commented Jul 7, 2015

AmplabJenkins commented Jul 7, 2015

mengxr commented Jul 7, 2015