Adding basic detect id columns guardrail #135
Conversation
Codecov Report
@@ Coverage Diff @@
## master #135 +/- ##
==========================================
+ Coverage 96.64% 96.65% +0.01%
==========================================
Files 89 90 +1
Lines 2233 2271 +38
==========================================
+ Hits 2158 2195 +37
- Misses 75 76 +1
Continue to review full report at Codecov.
|
| A dictionary of features with column name or index and their probability of being ID columns | ||
| """ | ||
| id_cols = {} | ||
| col_names = [str(col) for col in X.columns.tolist()] |
There was a problem hiding this comment.
I find the logic internally here a bit a hard to follow.
it seems to me that if
.95 if any one of the 3 cases are true
or
1.0 if case 1 and 2 are true or 2 and 3 are true (case 1 and 3 being true isn't possible, but it's not immediately obvious through reading).
maybe we can take another stab to refactor? happy to discuss more if needed
There was a problem hiding this comment.
Related to my comment on parameters about being more generous; since we're just issuing warnings would it be better to just set to 1.0 if any of the cases are true?
There was a problem hiding this comment.
Given the current checks, it may make sense to do as you suggested @jeremyliweishih, as each of the checks are decent indications of an ID column... Otherwise, I could give each check a "confidence percentage" and sum up a column's percentage across the three current checks. Thoughts?
There was a problem hiding this comment.
i think for now, let's not worry about the implementation much. as long as we're happy with API, we can change implementation in the future
| A dictionary of features with column name or index and their probability of being ID columns | ||
| """ | ||
| id_cols = {} | ||
| col_names = [str(col) for col in X.columns.tolist()] |
There was a problem hiding this comment.
i think for now, let's not worry about the implementation much. as long as we're happy with API, we can change implementation in the future
Fixes #115