Update cfm to use nullable types when needed - #1810
Conversation
thehomebrewnerd
changed the title
6b771aa to
a463433
Compare
thehomebrewnerd
changed the title
| # Use the same columns and make sure both indexes are sorted the same | ||
| dask_computed_fm = dask_fm.compute().set_index('id').loc[fm.index][fm.columns] | ||
| pd.testing.assert_frame_equal(fm, dask_computed_fm) | ||
| pd.testing.assert_frame_equal(fm, dask_computed_fm, check_dtype=False) |
There was a problem hiding this comment.
I updated these test checks to ignore the dtypes, because this update means we can no longer expect the dask and koalas feature matrices to have the same dtypes for all the columns. With pandas, the logical types would only change when they need to, but with Dask and Koalas they would always end up using the nullable types in the feature matrix.
Codecov Report
@@ Coverage Diff @@
## main #1810 +/- ##
==========================================
+ Coverage 98.36% 98.79% +0.42%
==========================================
Files 142 142
Lines 15820 15850 +30
==========================================
+ Hits 15562 15659 +97
+ Misses 258 191 -67
Continue to review full report at Codecov.
|
|
Have we done a performance test run on the branch? |
|
@rwedge Just now launched a run to compare commit |
Pandas performance didn't change much with this update. |
tamargrey
left a comment
tamargrey
left a comment
There was a problem hiding this comment.
LGTM! Also I confirmed these changes allow us to use a Lag primitive where nans are introduced on columns that have non nullable logical types!
| on: | ||
| schedule: | ||
| - cron: '*/30 * * * *' | ||
| - cron: '*/5 * * * *' |
There was a problem hiding this comment.
I asked Nate for this change (rather than putting up a 1 line PR). I want this workflow to run more often.
@tamargrey Nice - thanks for checking that out! |
4 participants
Update cfm to use nullable types when needed
Closes #1692
Updates the CFM process to use Woodwork nullable logical types when needed, instead of the non nullable types. This is important for situations where the cutoff time settings cause null values to be introduced into the feature matrix for columns that previously did not have null values present.