Handle incoming Object dtype data #1645
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1645 +/- ##
=======================================
Coverage 98.78% 98.78%
=======================================
Files 98 98
Lines 11653 11728 +75
=======================================
+ Hits 11511 11586 +75
Misses 142 142
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
woodwork/type_sys/type_system.py
Outdated
| @@ -93,7 +93,7 @@ | |||
|
|
|||
| DEFAULT_TYPE = Unknown | |||
|
|
|||
| INFERENCE_SAMPLE_SIZE = 100000 | |||
| INFERENCE_SAMPLE_SIZE = 10_000 | |||
There was a problem hiding this comment.
Shouldn't this be an integer with no underscore?
| INFERENCE_SAMPLE_SIZE = 10_000 | |
| INFERENCE_SAMPLE_SIZE = 10000 |
There was a problem hiding this comment.
Decided to follow PEP 515 for this since it makes it easier to read
| coeff = np.abs(skew(series)) | ||
| except ValueError: | ||
| # skew can't handle Int64 dtype | ||
| coeff = np.abs(skew(series.astype("float64"))) |
There was a problem hiding this comment.
This was the cause of the current LG ww perf issue
| @@ -93,7 +93,7 @@ | |||
|
|
|||
| DEFAULT_TYPE = Unknown | |||
|
|
|||
| INFERENCE_SAMPLE_SIZE = 100000 | |||
| INFERENCE_SAMPLE_SIZE = 100_000 | |||
There was a problem hiding this comment.
We might have to keep this at 100,000 for the time being. Reducing this exposes issues with larger datasets like zillow which has a column that gets inferred as IntegerNullable but actually has a float in one of its >90,000 observations. Attempting to cast this as Int64 throws an error.
ParthivNaresh
changed the title
ParthivNaresh
requested review from
bchen1116,
Cmancuso,
jeff-hernandez and
simha104
Fixes: #1646, #1647
Changes made:
objectdtypedata. Perf tests for Woodwork and EvalML here.Int64dtype which also fixes the current Woodwork perf test blockerTo prevent holding up this PR any further, I'll be including perf tests for potential inference sampling changes at another time.