Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-15040][ML][PYSPARK] Add Imputer to PySpark #17316

Closed
wants to merge 5 commits into from
Closed

[SPARK-15040][ML][PYSPARK] Add Imputer to PySpark #17316

wants to merge 5 commits into from

Conversation

MLnick
Copy link
Contributor

@MLnick MLnick commented Mar 16, 2017

Add Python wrapper for Imputer feature transformer.

How was this patch tested?

New doc tests and tweak to PySpark ML tests.py

@MLnick
Copy link
Contributor Author

MLnick commented Mar 16, 2017

cc @hhbyyh

@SparkQA
Copy link

SparkQA commented Mar 16, 2017

Test build #74667 has finished for PR 17316 at commit 5efe889.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class Imputer(JavaEstimator, HasInputCols, JavaMLReadable, JavaMLWritable):
  • class ImputerModel(JavaModel, JavaMLReadable, JavaMLWritable):

@SparkQA
Copy link

SparkQA commented Mar 16, 2017

Test build #74669 has finished for PR 17316 at commit 5e53e05.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 16, 2017

Test build #74672 has finished for PR 17316 at commit 325c9cf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

hhbyyh
hhbyyh approved these changes Mar 16, 2017
View reviewed changes
Copy link
Contributor

@hhbyyh hhbyyh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

python/pyspark/ml/feature.py Outdated
.. note:: Experimental

Imputation estimator for completing missing values, either using the mean or the median
of the column in which the missing values are located. The input column should be of
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Shall we change all the "column" to "columns" since we are supporting multiple columns now...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do for Python and Scala doc

python/pyspark/ml/feature.py
"""
Gets the value of :py:attr:`outputCols` or its default value.
"""
return self.getOrDefault(self.outputCols)
Copy link
Contributor

@hhbyyh hhbyyh Mar 16, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me we should add

    require(get(inputCols).isDefined, "Input cols must be defined first.")
    require(get(outputCols).isDefined, "Output cols must be defined first.")

in transformschema

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need that? The first call to $(inputCols) in validateAndTransformSchema will just throw an error with Failed to find a default value ...

@hhbyyh hhbyyh mentioned this pull request Mar 16, 2017
Closed
@SparkQA
Copy link

SparkQA commented Mar 20, 2017

Test build #74874 has finished for PR 17316 at commit 5c272b5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 22, 2017

Test build #75012 has finished for PR 17316 at commit 7fd17dd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MLnick
Copy link
Contributor Author

MLnick commented Mar 24, 2017

Merged to master.

@asfgit asfgit closed this in d9f4ce6 Mar 24, 2017
asfgit pushed a commit that referenced this pull request Apr 3, 2017
@hhbyyh
[SPARK-19969][ML] Imputer doc and example
4d28e84 
## What changes were proposed in this pull request?

Add docs and examples for spark.ml.feature.Imputer. Currently scala and Java examples are included. Python example will be added after #17316

## How was this patch tested?

local doc generation and example execution

Author: Yuhao Yang <yuhao.yang@intel.com>

Closes #17324 from hhbyyh/imputerdoc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hhbyyh hhbyyh hhbyyh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@MLnick @SparkQA @hhbyyh