Add clean missing data module #22
Conversation
|
@imatiach-msft, |
|
LGTM, let's remember to add example notebook once this gets into build. |
Pass! — The build has succeeded.
| createMockDataset | ||
| } | ||
|
|
||
| override def schemaForDataset: StructType = ??? |
There was a problem hiding this comment.
Why leave this unimplemented?
| @@ -121,6 +121,18 @@ trait HasOutputCol extends Wrappable { | |||
| def getOutputCol: String = $(outputCol) | |||
| } | |||
|
|
|||
| trait HasInputCols extends Wrappable { | |||
There was a problem hiding this comment.
This is great thanks but we should be very careful about versioning Core now that we are public. Is there a way to add a @since tag or would that be too superfluous?
There was a problem hiding this comment.
yes, we can definitely do this later
There was a problem hiding this comment.
sure, we can do this later
|
discussed, we can add the "since" tag later, we should add it to all APIs, similar to Spark |
imatiach-msft
force-pushed
the
ilmat/missing-value-cleaner
branch
2 times, most recently
from
ee412f5
to
38abb47
Compare
|
Updated PR - added support for string, boolean types for custom value and added test cases |
elibarzilay
force-pushed
the
ilmat/missing-value-cleaner
branch
from
38abb47
to
44a495e
Compare
|
[ MMLSpark 0.5.dev14+1.g44a495e
Copy the link to this HDInsight Script Action to setup this build on a cluster. This is a build for github PR #22Associated Changes
|
Add clean missing data module with modes:
1.) Replace with mean - replaces missings with mean of fit column
2.) Replace with median - replaces missings with approximate median of fit column
3.) Replace with custom value - replaces missings with custom value specified by user
As well as several smoke tests
Also added HasInputCols and HasOutputCols as traits for multiple columns (which will be needed when imputation will be added).
Resolved issue: Add support for missing value cleaner #11