Allow boolean columns to be included in remove_highly_correlated_features#2231
Conversation
Codecov Report
@@ Coverage Diff @@
## main #2231 +/- ##
=======================================
Coverage 99.29% 99.29%
=======================================
Files 146 146
Lines 17577 17590 +13
=======================================
+ Hits 17453 17466 +13
Misses 124 124
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
| if pd.api.types.is_bool_dtype(less_complex_col): | ||
| less_complex_col = less_complex_col.astype("float64") | ||
|
|
||
| if abs(more_complex_col.corr(less_complex_col)) >= pct_corr_threshold: |
There was a problem hiding this comment.
Noob question but is it a normal case for corr to be nan?
There was a problem hiding this comment.
It's possible for the column type to LatLong or a logical type where you cannot calculate the correlation value.
There was a problem hiding this comment.
Makes sense. Also, I didn't realize that the feature matrix had columns of all zeros (like bools = diff_ints), which makes correlation being nan make perfect sense.
| assert f_name in matrix_columns | ||
|
|
||
|
|
||
| def test_remove_highly_correlatied_features_on_boolean_cols(): |
There was a problem hiding this comment.
| def test_remove_highly_correlatied_features_on_boolean_cols(): | |
| def test_remove_highly_correlated_features_on_boolean_cols(): |
tamargrey
force-pushed
the
fix-corr-on-bools
branch
from
7200f03 to
e61b05b
Compare
rwedge
left a comment
rwedge
left a comment
There was a problem hiding this comment.
Looks good
Can we squeeze in a docs fix? On line 21 of the release notes, instead of "user:`sbadithe`" it should be ":user:`sbadithe`"
4 participants
Fixes #2229
Converts boolean columns to
float64inremove_highly_correlated_featuresto avoid the error that comes from callingcorron boolean columns