One Hot Encoder: Drop one redundant feature by default for features with two categories#1997
Merged
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1997 +/- ##
=========================================
+ Coverage 100.0% 100.0% +0.1%
=========================================
Files 274 274
Lines 22325 22360 +35
=========================================
+ Hits 22319 22354 +35
Misses 6 6
Continue to review full report at Codecov.
|
angela97lin
commented
Mar 21, 2021
freddyaboulton
approved these changes
Mar 22, 2021
freddyaboulton
left a comment
Contributor
There was a problem hiding this comment.
@angela97lin Nice! This was a surprisingly tricky one. I think this looks good. I have some minor non-blocking comments and a question on test_ohe_column_names_unique
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1936
Logic for our own custom dropping for binary features (features with only two categories):
is_binary, determine which features have two categories and which is the majority feature which should be dropped. Store this information. PassNoneto scikit-learn OHE so that scikit-learn does not drop.get_feature_names, find the name of the column that we should drop. Ex: col "original" with majority category "a" might have a transformed name of "original_a"; store "original_a" so we know to drop this column. Before returning, drop all columns that have been specified.Impl notes:
_get_feature_namesfromget_feature_names, where_get_feature_namesis pre-dropping binary features and more for our private implementation, whileget_feature_namesis user-facing and has the expected cols (without the binary feature that is dropped)