Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update variable type inference to better check for string values #683

Merged
merged 7 commits into from Jul 24, 2019

Conversation

@kmax12
Copy link
Member

commented Jul 23, 2019

Fixes a small issue that came up in pandas 0.25.0 testing.

While fixing, also did a small refactor to how we infer variable types

@kmax12 kmax12 changed the title Update infer to better check for string values Update variable type inference to better check for string values Jul 23, 2019

@@ -39,20 +39,28 @@ def infer_variable_types(df, link_vars, variable_types, time_index, secondary_ti
else:
inferred_type = vtypes.Numeric

elif variable in link_vars:
inferred_type = vtypes.Categorical

This comment has been minimized.

Copy link
@kmax12

kmax12 Jul 23, 2019

Author Member

link_vars should be categorical, not ordinal.

# heuristics to predict this some other than categorical
sample = df[variable].sample(min(10000, len(df[variable])))

# catch cases where object dtype cannot be interpreted as a string

This comment has been minimized.

Copy link
@kmax12

kmax12 Jul 23, 2019

Author Member

this is the core of the fix

return False
else:
return True
# if it can be casted to numeric, it's not a datetime

This comment has been minimized.

Copy link
@kmax12

kmax12 Jul 23, 2019

Author Member

addressed the todo, to specifically throw away datetimes that are numeric

# finally, try to cast to datetime
if col.dtype.name.find('str') > -1 or col.dtype.name.find('object') > -1:
try:
pd.to_datetime(col.dropna(), errors='raise')

This comment has been minimized.

Copy link
@kmax12

kmax12 Jul 23, 2019

Author Member

updated to just check all columns

@kmax12 kmax12 requested a review from rwedge Jul 23, 2019

@codecov

This comment has been minimized.

Copy link

commented Jul 23, 2019

Codecov Report

Merging #683 into master will increase coverage by 0.02%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #683      +/-   ##
==========================================
+ Coverage   97.44%   97.47%   +0.02%     
==========================================
  Files         118      118              
  Lines        9643     9670      +27     
==========================================
+ Hits         9397     9426      +29     
+ Misses        246      244       -2
Impacted Files Coverage Δ
featuretools/computational_backends/feature_set.py 100% <ø> (ø) ⬆️
...eaturetools/tests/utils_tests/test_entity_utils.py 100% <100%> (ø) ⬆️
featuretools/utils/entity_utils.py 98.03% <100%> (+2.24%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f4ffd7...1bbe25b. Read the comment docs.

kmax12 added 5 commits Jul 23, 2019
@rwedge
rwedge approved these changes Jul 24, 2019
Copy link
Contributor

left a comment

Looks good!

@kmax12 kmax12 merged commit eabad43 into master Jul 24, 2019

2 checks passed

license/cla Contributor License Agreement is signed.
Details
test_all_python_versions Workflow: test_all_python_versions
Details
@rwedge rwedge referenced this pull request Aug 19, 2019

@kmax12 kmax12 deleted the integer-object-dtype-infer-variable branch Sep 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.