Skip to content

[SPARK-13786][ML][PYTHON] Removed save/load for python tuning#12782

Closed
jkbradley wants to merge 1 commit intoapache:masterfrom
jkbradley:remove-python-tuning-saveload
Closed

[SPARK-13786][ML][PYTHON] Removed save/load for python tuning#12782
jkbradley wants to merge 1 commit intoapache:masterfrom
jkbradley:remove-python-tuning-saveload

Conversation

@jkbradley
Copy link
Member

What changes were proposed in this pull request?

Per discussion on [https://github.com//pull/12604], this removes ML persistence for Python tuning (TrainValidationSplit, CrossValidator, and their Models) since they do not handle nesting easily. This support should be re-designed and added in the next release.

How was this patch tested?

Removed unit test elements saving and loading the tuning algorithms, but kept tests to save and load their bestModel fields.

@jkbradley
Copy link
Member Author

Rebased to fix conflicts

@SparkQA
Copy link

SparkQA commented Apr 29, 2016

Test build #57352 has finished for PR 12782 at commit d70bfcb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class CrossValidator(Estimator, ValidatorParams):
    • class CrossValidatorModel(Model, ValidatorParams):
    • class TrainValidationSplit(Estimator, ValidatorParams):
    • class TrainValidationSplitModel(Model, ValidatorParams):

@SparkQA
Copy link

SparkQA commented Apr 29, 2016

Test build #57351 has finished for PR 12782 at commit a91f39e.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds the following public classes (experimental):
    • class CrossValidator(Estimator, ValidatorParams):
    • class CrossValidatorModel(Model, ValidatorParams):
    • class TrainValidationSplit(Estimator, ValidatorParams):
    • class TrainValidationSplitModel(Model, ValidatorParams):

@jkbradley jkbradley force-pushed the remove-python-tuning-saveload branch from d70bfcb to 6df120f Compare April 29, 2016 21:00
@jkbradley
Copy link
Member Author

Rebased to fix apparent unclean merge

@SparkQA
Copy link

SparkQA commented Apr 29, 2016

Test build #57361 has finished for PR 12782 at commit 6df120f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class CrossValidator(Estimator, ValidatorParams):
    • class CrossValidatorModel(Model, ValidatorParams):
    • class TrainValidationSplit(Estimator, ValidatorParams):
    • class TrainValidationSplitModel(Model, ValidatorParams):

@jkbradley
Copy link
Member Author

@yinxusen Would you mind taking a look? Thank you!

from pyspark.ml import Estimator, Model, Pipeline, PipelineModel, Transformer
from pyspark.ml.classification import (
LogisticRegression, DecisionTreeClassifier, OneVsRest, OneVsRestModel)
from pyspark.ml.classification import *
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you need to have a wildcard import?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are ~5 imports, so I think it's worthwhile.

@thunterdb
Copy link
Contributor

thunterdb commented Apr 30, 2016

@jkbradley this looks good to me

@mengxr
Copy link
Contributor

mengxr commented Apr 30, 2016

Merged into master. Thanks!

@asfgit asfgit closed this in 09da43d Apr 30, 2016
@jkbradley
Copy link
Member Author

Thanks!

@jkbradley jkbradley deleted the remove-python-tuning-saveload branch April 30, 2016 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants