[SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini #17407

facaiy · 2017-03-24T04:25:27Z

Fix bug: DecisionTreeModel can't recongnize Impurity "Gini" when loading

TODO:

add unit test
fix the bug

jkbradley · 2017-03-24T21:48:21Z

Thanks for catching this bug! I recommend we not modify the value when the setter is called, but rather convert to lowercase internally. In this case, it looks like we can just change ImpurityCalculator.getCalculator.

SparkQA · 2017-03-25T10:55:20Z

Test build #3611 has finished for PR 17407 at commit 604dcd1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

facaiy · 2017-03-25T11:06:14Z

@jkbradley Thanks. I agree with your advice. Modifying the value is a little aggressive, while changing ImpurityCalculator.getCalculator is more moderate. However, I'm afraid that the similar bugs, because of mismatch, won't leave if we don't fix it at first.

Anyway, I'd like to take a try according to your advice.
However, some unit tests are broken, though I have tested them before pushing. Perhaps there is some wrong with my IDE, and I need to take a check at first on next Monday.

jkbradley · 2017-03-26T00:05:21Z

I agree it'd be nice to come up with a generic fix for making String Params robust to case, but I don't have a good solution right now. I'll think about how we might put some generic testing in place...

jkbradley

Thanks for the updates! My comments are just for small cleanups.

jkbradley · 2017-03-26T00:07:13Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

+    val rdd = TreeTests.getTreeReadWriteData(sc)
+
+    val categoricalData: DataFrame =
+      TreeTests.setMetadata(rdd, Map(0 -> 2, 1 -> 3), numClasses = 2)


To simplify this, you can write TreeTests.setMetadata(rdd, Map.empty[Int, Int], numClasses = 2).
Since this is not testing categorical features, there's no need to throw them in.

jkbradley · 2017-03-26T00:07:14Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

+    val categoricalData: DataFrame =
+      TreeTests.setMetadata(rdd, Map(0 -> 2, 1 -> 3), numClasses = 2)
+
+    // BUG: see SPARK-20043


I'd put the JIRA number in the test title.

removed the comment.

jkbradley · 2017-03-26T00:08:10Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

@@ -385,6 +385,20 @@ class DecisionTreeClassifierSuite
    testEstimatorAndModelReadWrite(dt, continuousData, allParamSettings ++ Map("maxDepth" -> 0),
      allParamSettings ++ Map("maxDepth" -> 0), checkModelData)
  }
+
+  test("read/write: ImpurityCalculator builder did not recognize impurity type: Gini") {


To be more specific, how about:
"SAPRK-20043: ImpurityCalculator builder fails for uppercase impurity type Gini in model read/write"

jkbradley · 2017-03-26T00:08:26Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

+      TreeTests.setMetadata(rdd, Map(0 -> 2, 1 -> 3), numClasses = 2)
+
+    // BUG: see SPARK-20043
+    val dt = new DecisionTreeClassifier().setImpurity("Gini")


To make this faster, set maxDepth = 2 (something small)

set maxDepth = 2.

SparkQA · 2017-03-26T00:56:52Z

Test build #3613 has finished for PR 17407 at commit 80f3306.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2017-03-26T02:17:18Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

@@ -385,6 +385,22 @@ class DecisionTreeClassifierSuite
    testEstimatorAndModelReadWrite(dt, continuousData, allParamSettings ++ Map("maxDepth" -> 0),
      allParamSettings ++ Map("maxDepth" -> 0), checkModelData)
  }
+
+  test("SAPRK-20043: " +


Sorry, I had a typo: SPARK, not SAPRK

oops, copy & paste. thanks for correction.

SparkQA · 2017-03-26T08:44:45Z

Test build #3616 has finished for PR 17407 at commit 55fe603.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

facaiy · 2017-03-26T10:31:18Z

@jkbradley Hi, tests passed. Is it good enough to be merged?

By the way, String Params are fragile when saving/loading model, as setParam and getParam methods are useless in such case.

hhbyyh · 2017-03-27T05:45:21Z

mllib/src/test/scala/org/apache/spark/ml/regression/DecisionTreeRegressorSuite.scala

+      .setMaxDepth(2)
+
+    val model = dt.fit(data)
+


the second unit test seems redundant for this case.

hhbyyh · 2017-03-27T05:45:30Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

+      .setMaxDepth(2)
+
+    val model = dt.fit(data)
+


The blank lines kinda stand out.

delete some blank lines to keep compact.

hhbyyh · 2017-03-27T05:46:58Z

mllib/src/test/scala/org/apache/spark/ml/classification/DecisionTreeClassifierSuite.scala

+
+    val model = dt.fit(data)
+
+    testDefaultReadWrite(model, false)


how about setting testParams=true for this case.

jkbradley · 2017-03-28T21:34:03Z

LGTM pending tests
Thank you!

SparkQA · 2017-03-28T22:27:36Z

Test build #3618 has finished for PR 17407 at commit 22ee03d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2017-03-28T23:13:29Z

Merging with master and branch-2.1

… for uppercase impurity type Gini Fix bug: DecisionTreeModel can't recongnize Impurity "Gini" when loading TODO: + [x] add unit test + [x] fix the bug Author: 颜发才（Yan Facai） <facai.yan@gmail.com> Closes #17407 from facaiy/BUG/decision_tree_loader_failer_with_Gini_impurity. (cherry picked from commit 7d432af) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>

facaiy force-pushed the BUG/decision_tree_loader_failer_with_Gini_impurity branch from 8951fe1 to 78aa380 Compare March 24, 2017 05:37

facaiy changed the title ~~[SPARK-20043][ML][WIP] DecisionTreeModel can't recongnize Impurity "Gini" when loading~~ [SPARK-20043][ML] DecisionTreeModel can't recongnize Impurity "Gini" when loading Mar 24, 2017

facaiy force-pushed the BUG/decision_tree_loader_failer_with_Gini_impurity branch from 604dcd1 to 05c998c Compare March 25, 2017 23:49

facaiy added 2 commits March 26, 2017 07:52

TST: add testsuite

b554fce

BUG: fix for loading exception

80f3306

facaiy force-pushed the BUG/decision_tree_loader_failer_with_Gini_impurity branch from 05c998c to 80f3306 Compare March 25, 2017 23:52

jkbradley reviewed Mar 26, 2017

View reviewed changes

TST: modify testsuite

276cac6

jkbradley reviewed Mar 26, 2017

View reviewed changes

TST: fix a typo

55fe603

facaiy changed the title ~~[SPARK-20043][ML] DecisionTreeModel can't recongnize Impurity "Gini" when loading~~ [SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type GiniImpurityCalculator builder fails for uppercase impurity type Gini Mar 26, 2017

hhbyyh reviewed Mar 27, 2017

View reviewed changes

facaiy added 2 commits March 28, 2017 09:59

TST: keep compact and open testParams

5e0fbe3

TST: remove redundant unit test

22ee03d

asfgit closed this in 7d432af Mar 28, 2017

facaiy deleted the BUG/decision_tree_loader_failer_with_Gini_impurity branch March 29, 2017 00:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini #17407

[SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini #17407

facaiy commented Mar 24, 2017 •

edited

Loading

jkbradley commented Mar 24, 2017

SparkQA commented Mar 25, 2017

facaiy commented Mar 25, 2017 •

edited

Loading

jkbradley commented Mar 26, 2017

jkbradley left a comment

jkbradley Mar 26, 2017

facaiy Mar 26, 2017

jkbradley Mar 26, 2017

facaiy Mar 26, 2017

jkbradley Mar 26, 2017

facaiy Mar 26, 2017

jkbradley Mar 26, 2017

facaiy Mar 26, 2017

SparkQA commented Mar 26, 2017

jkbradley Mar 26, 2017

facaiy Mar 26, 2017

SparkQA commented Mar 26, 2017

facaiy commented Mar 26, 2017

hhbyyh Mar 27, 2017

facaiy Mar 28, 2017

hhbyyh Mar 27, 2017

facaiy Mar 28, 2017

hhbyyh Mar 27, 2017

facaiy Mar 28, 2017

jkbradley commented Mar 28, 2017

SparkQA commented Mar 28, 2017

jkbradley commented Mar 28, 2017

[SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini #17407

[SPARK-20043][ML] DecisionTreeModel: ImpurityCalculator builder fails for uppercase impurity type Gini #17407

Conversation

facaiy commented Mar 24, 2017 • edited Loading

jkbradley commented Mar 24, 2017

SparkQA commented Mar 25, 2017

facaiy commented Mar 25, 2017 • edited Loading

jkbradley commented Mar 26, 2017

jkbradley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 26, 2017

facaiy commented Mar 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkbradley commented Mar 28, 2017

SparkQA commented Mar 28, 2017

jkbradley commented Mar 28, 2017

facaiy commented Mar 24, 2017 •

edited

Loading

facaiy commented Mar 25, 2017 •

edited

Loading