Allow AutoMLSearch to handle Unknown type #2477

bchen1116 · 2021-07-08T19:08:02Z

Perf tests here

codecov · 2021-07-08T19:11:21Z

Codecov Report

Merging #2477 (bbc1021) into main (b892fc9) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2477     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        283     283             
  Lines      25808   25897     +89     
=======================================
+ Hits       25707   25796     +89     
  Misses       101     101

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`99.4% <100.0%> (+0.1%)`	⬆️
evalml/model_understanding/graphs.py	`100.0% <100.0%> (ø)`
evalml/pipelines/utils.py	`99.2% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`99.8% <100.0%> (+0.1%)`	⬆️
evalml/tests/component_tests/test_lsa.py	`100.0% <100.0%> (ø)`
...l/tests/component_tests/test_per_column_imputer.py	`100.0% <100.0%> (ø)`
...alml/tests/component_tests/test_text_featurizer.py	`100.0% <100.0%> (ø)`
evalml/tests/conftest.py	`98.3% <100.0%> (+0.1%)`	⬆️
evalml/tests/data_checks_tests/test_data_checks.py	`100.0% <100.0%> (ø)`
...ecks_tests/test_natural_language_nan_data_check.py	`100.0% <100.0%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b892fc9...bbc1021. Read the comment docs.

bchen1116 · 2021-07-12T22:04:32Z

evalml/tests/component_tests/test_per_column_imputer.py

    for logical_type in override_types:
        # Column with Nans to boolean used to fail. Now it doesn't
-        if has_nan and logical_type == Boolean:
+        if has_nan and logical_type in [Boolean, NaturalLanguage]:


Leave out NaturalLanguage since casting this will result in np.nan becoming pd.NA, which fails the imputer

But how come this wouldn't happen before? Woodwork doesn't convert np.nan to pd.NA in 0.4.2? Is this because of the pandas upgrade?

@freddyaboulton I believe it's due to the pandas upgrade!

Looking at the 1.3.0 release docs, seems like there's a lot of changes with NaN handling, and they're using <NA> for scalar types

Thanks for looking into this @bchen1116 Let's list this as a breaking change for now. I imagine we might want to file an issue to discuss if there are any changes we need to make to the simpleimputer? If users run it on natural language after this pr they'll get a stacktrace they didn't get before.

@freddyaboulton updated the release notes with the breaking change and filed the issue here!

bchen1116 · 2021-07-13T16:59:43Z

evalml/tests/data_checks_tests/test_natural_language_nan_data_check.py

-    assert nl_nan_check.validate([nl_col, nl_col_without_nan]) == expected
-
-    # test np.array
-    assert nl_nan_check.validate(np.array([nl_col, nl_col_without_nan])) == expected


Since we can't cast this to NL type without ww, we remove this from the tests. They'll be cast as Unknown

…426_unknown

freddyaboulton

@bchen1116 Thank you for doing this! The changes look good to me and I'm glad the performance looks better with this change for a lot of the datasets we commonly test with.

One thing I'd like to find out/cover with (small/fast) unit tests before merge though - Do our model understanding methods support Unknown features?

I think we should focus on

partial dependence
permutation importance
prediction explanations

My hope is yes but I'm not 100% sure.

freddyaboulton · 2021-07-14T18:06:24Z

evalml/tests/component_tests/test_per_column_imputer.py

    for logical_type in override_types:
        # Column with Nans to boolean used to fail. Now it doesn't
-        if has_nan and logical_type == Boolean:
+        if has_nan and logical_type in [Boolean, NaturalLanguage]:


But how come this wouldn't happen before? Woodwork doesn't convert np.nan to pd.NA in 0.4.2? Is this because of the pandas upgrade?

evalml/tests/automl_tests/test_automl.py

evalml/tests/component_tests/test_text_featurizer.py

chukarsten

A nit here and there, but overall, I like where this is headed. Some of the things I'd like for you to consider, particularly with respect to the explanation predictions, partial dependence and permutation importance is whether we're adding back in more run time? I need to reacquaint myself with Freddy's testing environment/context manager to see whether that could help us there, but I'd just like to be cognizant of whether we're walking back any of his work by adding in these tests.

Overall, though, this is solid. Thanks a lot for addressing it so quickly from design to impl and getting it out there.

evalml/automl/automl_search.py

evalml/model_understanding/graphs.py

evalml/pipelines/utils.py

chukarsten · 2021-07-19T14:35:35Z

evalml/tests/model_understanding_tests/test_partial_dependence.py

+            partial_dependence(pl, X, indices, grid_resolution=10)
+        return
+    s = partial_dependence(pl, X, indices, grid_resolution=10)


How low can we keep this grid_resolution?

Lowest is 2

bchen1116 · 2021-07-19T16:19:01Z

@chukarsten two of the tests take about 1 second, while the permutation importance test add about 3 seconds. I think it's important to have the coverage on these tests, and it doesn't take too long to raise concern for now imo. Let me know what you think!

handle woodwork unknown

09944f3

bchen1116 self-assigned this Jul 8, 2021

update release notes

7cf15e4

bchen1116 added 14 commits July 8, 2021 12:29

update ww version

f31bf9e

fix deps

285e125

Merge branch 'main' into bc_2426_unknown

83bbaa3

update reqs

63c16c9

fixing tests

422ee1c

Merge branch 'main' into bc_2426_unknown

0eeff5e

Merge branch 'main' into bc_2426_unknown

515e818

update docs and clean tests

0260c6f

Merge branch 'main' into bc_2426_unknown

0391219

fixing test

f6a179f

Merge branch 'main' into bc_2426_unknown

e063860

Merge branch 'main' into bc_2426_unknown

1009baf

update deps

7e70340

add line

99ec0ac

bchen1116 commented Jul 12, 2021

View reviewed changes

bchen1116 commented Jul 13, 2021

View reviewed changes

bchen1116 added 4 commits July 13, 2021 10:01

updat deps

ae36364

Merge branch 'main' into bc_2426_unknown

d2d760e

update dependencies

2bcb251

Merge branch 'bc_2426_unknown' of github.com:alteryx/evalml into bc_2…

5dd3480

…426_unknown

bchen1116 marked this pull request as ready for review July 13, 2021 17:08

bchen1116 requested review from angela97lin, chukarsten and dsherry and removed request for dsherry July 13, 2021 17:08

bchen1116 requested review from ParthivNaresh, christopherbunn, freddyaboulton and jeremyliweishih July 13, 2021 17:09

bchen1116 added 3 commits July 13, 2021 12:40

Merge branch 'main' into bc_2426_unknown

6aac33b

fixing deps

78233b1

Merge branch 'main' into bc_2426_unknown

837cb0f

freddyaboulton approved these changes Jul 14, 2021

View reviewed changes

bchen1116 added 4 commits July 14, 2021 15:02

adding test coverage

4cd8368

Merge branch 'main' into bc_2426_unknown

6c8d7c4

update breaking changes

531e447

fixing formatting

c3778a3

chukarsten approved these changes Jul 19, 2021

View reviewed changes

address comments

0472355

bchen1116 added 6 commits July 19, 2021 09:24

string to regex

ea38c79

address test failures

ebefee5

update dep order

93363f8

Merge branch 'main' into bc_2426_unknown

3ad383b

Merge branch 'main' into bc_2426_unknown

9c18416

Merge branch 'main' into bc_2426_unknown

bbc1021

chukarsten merged commit 87df494 into main Jul 20, 2021

chukarsten deleted the bc_2426_unknown branch July 20, 2021 17:53

chukarsten mentioned this pull request Jul 22, 2021

v0.29.0 #2536

Merged

Allow AutoMLSearch to handle Unknown type #2477

Allow AutoMLSearch to handle Unknown type #2477

Uh oh!

Conversation

bchen1116 commented Jul 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jul 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bchen1116 Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton left a comment • edited by bchen1116 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bchen1116 commented Jul 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bchen1116 commented Jul 8, 2021 •

edited

Loading

codecov bot commented Jul 8, 2021 •

edited

Loading

bchen1116 Jul 14, 2021 •

edited

Loading

freddyaboulton left a comment •

edited by bchen1116

Loading