[SPARK-10188] [Pyspark] Pyspark CrossValidator with RMSE selects incorrect model #8399
Conversation
|
ok to test |
|
Thanks for the PR! I'll take a look now. |
| @@ -66,6 +66,9 @@ def evaluate(self, dataset, params=None): | |||
| else: | |||
| raise ValueError("Params must be a param map but got %s." % type(params)) | |||
|
|
|||
| def isLargerBetter(self): | |||
There was a problem hiding this comment.
Can you please copy the doc from Scala here? (no need to copy to child classes since the "inherit_doc" tag will handle that)
|
Looks good except for those 2 items. |
|
Test build #1695 has finished for PR 8399 at commit
|
|
Test build #1699 has finished for PR 8399 at commit
|
|
Ping @mengxr In case I can't check this soon, it would be great to get this into 1.5 if there is an RC3. |
|
Test build #1700 has finished for PR 8399 at commit
|
|
That would be great - I've just messaged him. If there are any other changes you need to get this into 1.5 I'll get them in ASAP today. |
|
No, this LGTM. I'll merge this with branch-1.5 and master now. Thanks very much! |
…rrect model * Added isLargerBetter() method to Pyspark Evaluator to match the Scala version. * JavaEvaluator delegates isLargerBetter() to underlying Scala object. * Added check for isLargerBetter() in CrossValidator to determine whether to use argmin or argmax. * Added test cases for where smaller is better (RMSE) and larger is better (R-Squared). (This contribution is my original work and that I license the work to the project under Sparks' open source license) Author: noelsmith <mail@noelsmith.com> Closes #8399 from noel-smith/pyspark-rmse-xval-fix. (cherry picked from commit 7583681) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>
(This contribution is my original work and that I license the work to the project under Sparks' open source license)