Support custom woodwork types in components by angela97lin · Pull Request #1784 · alteryx/evalml

angela97lin · 2021-02-02T23:12:03Z

Closes #1662. Another large-ish PR but not too much logic change--just replacing the final line to convert to WW to new util method, and writing a lot of tests. I initially wanted to iterate through all the components and test like the pattern we already have in test_components.py but realized that each component has its own nuanced behavior, so decided against it and added the tests to their individual files. The parameterization is similar in many cases though. 😬

Moves Woodwork-related util methods to woodwork_utils.py and move tests to test_woodwork_utils.py
Add private utility method to help best reassign logical types after transformation.
Merged _convert_to_woodwork_structure with infer_feature_types by combining the two implementations.

Notes on compatible and incompatible types. Mostly for my own understanding, but also helped inform test cases.

Datetime

Datetime --> Double: errors
Datetime --> Integer: errors
Datetime --> Categorical: works
Datetime --> Natural Language: works

Categorical

Categorical --> Datetime: errors
Categorical --> Double: works, sometimes. ("1", "2" okay, "a", "b" not okay)
Categorical --> Natural Language: works
Categorical --> Integer: errors

Integer

Integer --> Datetime: works... but isn't great lol (literally .000000001 ms)
Integer --> Double: works
Integer --> Categorical: works
Integer --> Natural Language: works

Double

Double --> Datetime: works... but isn't great lol (literally .000000001 ms)
Double --> Integer: works sometimes (if possible to convert)
Double --> Categorical: works
Double --> Natural Language: works

**Natural Language **

Natural Language --> Datetime: errors
Natural Language --> Double: works, sometimes
Natural Language --> Categorical: works
Natural Language --> Integer: works, sometimes
There's no good way of confirming when it is safe to convert floats to category or when it is not without trying. For example: can convert ['1', '2'] (categories) to float, but not ['a', 'b']. Hence, current implementation will try to set the type. If it fails, then don't do anything. If it succeeds, add to logical type dictionary to pass along to the initialization of Woodwork DataTable, so that Woodwork does not have to do type inference, saving us some time.
Look into: featuretools component which changed the columns of the original input DF?

codecov · 2021-02-03T00:07:05Z

Codecov Report

Merging #1784 (ad57157) into main (5cba8a9) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1784     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         252      255      +3     
  Lines       20250    20564    +314     
=========================================
+ Hits        20242    20556    +314     
  Misses          8        8

Impacted Files	Coverage Δ
evalml/tests/utils_tests/test_gen_utils.py	`100.0% <ø> (ø)`
evalml/utils/gen_utils.py	`99.6% <ø> (-<0.1%)`	⬇️
evalml/automl/automl_search.py	`99.7% <100.0%> (-<0.1%)`	⬇️
evalml/data_checks/class_imbalance_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/data_checks.py	`100.0% <100.0%> (ø)`
evalml/data_checks/highly_null_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/id_columns_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/invalid_targets_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/multicollinearity_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/no_variance_data_check.py	`100.0% <100.0%> (ø)`
... and 70 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5cba8a9...ad57157. Read the comment docs.

…yx/evalml into 1662_custom_woodwork_in_components

angela97lin · 2021-02-04T19:23:29Z

    drop_rows_with_nans
    infer_feature_types
    save_plot
+    is_all_numeric


Unrelated change but adding to API ref :)

angela97lin · 2021-02-04T20:07:18Z


 logger = get_logger(__file__)

-numeric_and_boolean_ww = [ww.logical_types.Integer, ww.logical_types.Double, ww.logical_types.Boolean]


Moved all Woodwork related utils to woodwork_utils.py :)

angela97lin · 2021-02-04T20:19:46Z

 @pytest.mark.parametrize("data_type", ['pd', 'ww'])
 def test_simple_imputer_bool_dtype_object(data_type, make_data_type):
-    X = pd.DataFrame([True, np.nan, False, np.nan, True], dtype=object)
+    X = pd.DataFrame([True, np.nan, False, np.nan, True], dtype='boolean')


This update (and many of the others made in the tests) was made because now we try to convert the columns back to the original types. Before, the original dtype was specified as object, so the column would have been stored as a categorical col and we would have expected X_expected_arr to return a categorical col. We need to explicitly set to boolean for the expected outcome to be boolean.

I think these conversions make sense over previous behavior--user input is boolean, so output is boolean.

Agreed! Glad we don't have to do this anymore.

angela97lin · 2021-02-04T21:04:01Z

        Returns:
            ww.DataTable: Transformed X
        """
+        X_ww = _convert_to_woodwork_structure(X)


I decided to move these out of the try/except because I think it's more clear what the try/except is for. If it errors out while doing Woodwork conversions (as it did while I was testing stuff), it shouldn't throw a MethodPropertyNotFoundError :)

angela97lin · 2021-02-08T04:21:32Z

    assert X_renamed.logical_types == {0: ww.logical_types.Categorical, 1: ww.logical_types.Categorical}


-def test_convert_woodwork_types_wrapper_with_nan():


Moved to test_woodwork_utils.py

freddyaboulton

@angela97lin Looks good to me! I have some questions on the unit tests but nothing blocking.

freddyaboulton · 2021-02-08T23:18:16Z

 @pytest.mark.parametrize("data_type", ['pd', 'ww'])
 def test_simple_imputer_bool_dtype_object(data_type, make_data_type):
-    X = pd.DataFrame([True, np.nan, False, np.nan, True], dtype=object)
+    X = pd.DataFrame([True, np.nan, False, np.nan, True], dtype='boolean')


Agreed! Glad we don't have to do this anymore.

freddyaboulton · 2021-02-08T23:36:04Z

+        transformed = dft.transform(X, y)
+        assert isinstance(transformed, ww.DataTable)
+        if logical_type == Integer or logical_type == Double or logical_type == Categorical:
+            assert transformed.logical_types == {0: logical_type,


I think we should modify the delayed feature transformer to carry over the ints. I'd be happy to file an issue if you are on board!

Sure, that sounds good! I'm guessing shift just returns floats which is why we end up with Doubles right now?

chukarsten

Oh man. Great work getting through all this and testing it all...what a feat of endurance. Just a few questions I had here and there. If you want to get back to me on them, just ping me on Slack so I can go check it out, but nothing to really stop merge.

chukarsten · 2021-02-10T15:41:34Z

-        X = _convert_to_woodwork_structure(X)
+        X = infer_feature_types(X)
        if y is not None:
-            y = _convert_to_woodwork_structure(y)
+            y = infer_feature_types(y)


That's my boy right there! I DREAM about this code chunk.

Honestly, I've seen this so often I swear I do too 😆

chukarsten · 2021-02-10T16:26:10Z

+        X_ww = infer_feature_types(X)
+        X = _convert_woodwork_types_wrapper(X_ww.to_dataframe())
+        if y is not None:
+            y = infer_feature_types(y)
+            y = _convert_woodwork_types_wrapper(y.to_series())


There it is <3

chukarsten · 2021-02-10T16:31:32Z

-        X = _convert_to_woodwork_structure(X)
-        cat_cols = list(X.select('category').columns)
+        X = infer_feature_types(X)
+        cat_cols = list(X.select(['category', 'boolean']).columns)


What prompted this? This seems like a fine change...but just curious. This comment is also a bookmark so I can go make sure this is tested.

I can double check we're testing this but IIRC, it's so that we treat boolean columns like categorical cols. We don't want to pass it to our numeric_imputer which might try to use "mean" as the impute strategy and end up with nonsense for the imputed values.

chukarsten · 2021-02-10T22:56:52Z

+        try:
+            X = ww.DataTable(X_df, logical_types={0: logical_type})
+        except TypeError:
+            continue


I think I'm tired, but what is the intent behind this chunk here?

Ah so basically, I'm trying every possible logical type conversion that's allowed (see desc of PR for allowed vs error), but rather than manually keeping track of all possible conversions, I'm just trying. Ex: Int --> Nat Lang is valid conversion and therefore a valid override a user might provide to EvalML for us to test. Nat Lang --> Int might not always be (and will raise a TypeError). If its not, don't try to pass to EvalML to test.

move utils and tests to woodwork_utils

1a0212f

angela97lin self-assigned this Feb 2, 2021

angela97lin added 2 commits February 2, 2021 18:21

oops, fix some imports

afae671

fix some imports

7c0bf12

angela97lin added 11 commits February 3, 2021 12:27

some inital testing

1b5116e

Merge branch 'main' into 1662_custom_woodwork_in_components

78a8a2a

update some components

dd64053

Merge branch '1662_custom_woodwork_in_components' of github.com:alter…

1fd9c85

…yx/evalml into 1662_custom_woodwork_in_components

update some more transformers

5da9c6c

oops fix standard scaler

d1bf5fa

update delayed feat transformer

0ca4496

update components cont

c9561c9

remove logic handling from component graph

c51f614

oops, delete from gen utils after moving

e939ecf

Merge branch 'main' into 1662_custom_woodwork_in_components

83e5abd

angela97lin commented Feb 4, 2021

View reviewed changes

uncomment test and fix docs

b308ac8

angela97lin commented Feb 4, 2021

View reviewed changes

update import statements and remove block abouyt physical type

dd8d2c2

angela97lin commented Feb 4, 2021

View reviewed changes

angela97lin added 5 commits February 4, 2021 16:04

some more cleanup

aa5ff71

revert

6aeb70b

add some tests

0cf1581

more messy testing

ec7612e

Merge branch 'main' into 1662_custom_woodwork_in_components

e149584

angela97lin commented Feb 8, 2021

View reviewed changes

angela97lin added 2 commits February 8, 2021 00:41

clean up some tests, more to go

d95e75c

fix some more tests and clean up

c181c3a

angela97lin added 4 commits February 8, 2021 16:42

update infer_feature_types

1a8c4a2

remove _convert_to_woodwork_structure

334c139

add test

f4f226d

update docs

06cf297

freddyaboulton approved these changes Feb 8, 2021

View reviewed changes

angela97lin added 2 commits February 9, 2021 12:01

Merge branch 'main' into 1662_custom_woodwork_in_components

2e853c3

Merge branch 'main' into 1662_custom_woodwork_in_components

3665785

angela97lin mentioned this pull request Feb 9, 2021

Add support for custom Woodwork types #1810

Open

angela97lin added 11 commits February 9, 2021 17:38

update tests

03f4405

Merge branch 'main' into 1662_custom_woodwork_in_components

5d9d06f

fix merge conflicts

234a846

oopsies, accidentally pushed testing change

1f8783e

update tests

4ceca88

update test for codecov

111b7d7

retrigger tests

5f5cc0b

fix test

3c4f9d1

Merge branch 'main' into 1662_custom_woodwork_in_components

238c51e

oopsies fix merge

064ced0

lint lint lint

8ae3511

angela97lin added this to the Sprint 2021 Feb A milestone Feb 10, 2021

chukarsten approved these changes Feb 10, 2021

View reviewed changes

angela97lin added 5 commits February 10, 2021 17:59

Merge branch 'main' into 1662_custom_woodwork_in_components

847d984

Merge branch 'main' into 1662_custom_woodwork_in_components

d3bca4f

Merge branch 'main' into 1662_custom_woodwork_in_components

8fc744a

address PR comments]

fd40185

merging main

ad57157

angela97lin merged commit d4e2f54 into main Feb 11, 2021

angela97lin deleted the 1662_custom_woodwork_in_components branch February 11, 2021 22:53

chukarsten mentioned this pull request Feb 23, 2021

v0.19.0 Release #1884

Merged

dsherry mentioned this pull request Mar 10, 2021

v0.20.0 #1961

Merged


		logger = get_logger(__file__)

		numeric_and_boolean_ww = [ww.logical_types.Integer, ww.logical_types.Double, ww.logical_types.Boolean]

		assert X_renamed.logical_types == {0: ww.logical_types.Categorical, 1: ww.logical_types.Categorical}


		def test_convert_woodwork_types_wrapper_with_nan():

Conversation

angela97lin commented Feb 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

angela97lin commented Feb 2, 2021 •

edited

Loading

codecov Bot commented Feb 3, 2021 •

edited

Loading