Upgrade Woodwork to 0.6.0 #2690

ParthivNaresh · 2021-08-24T16:52:45Z

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

codecov · 2021-08-24T16:57:13Z

Codecov Report

Merging #2690 (61c91ee) into main (635f3e7) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2690     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        301     301             
  Lines      27607   27678     +71     
=======================================
+ Hits       27563   27634     +71     
  Misses        44      44

Impacted Files	Coverage Δ
...alml/model_understanding/permutation_importance.py	`100.0% <100.0%> (ø)`
...elines/components/transformers/column_selectors.py	`100.0% <100.0%> (ø)`
...rs/preprocessing/transform_primitive_components.py	`100.0% <100.0%> (ø)`
...mponent_tests/test_column_selector_transformers.py	`100.0% <100.0%> (ø)`
...mponent_tests/test_delayed_features_transformer.py	`100.0% <100.0%> (ø)`
evalml/tests/component_tests/test_imputer.py	`100.0% <100.0%> (ø)`
...alml/tests/component_tests/test_lgbm_classifier.py	`100.0% <100.0%> (ø)`
...valml/tests/component_tests/test_lgbm_regressor.py	`100.0% <100.0%> (ø)`
...alml/tests/component_tests/test_one_hot_encoder.py	`100.0% <100.0%> (ø)`
...l/tests/component_tests/test_per_column_imputer.py	`100.0% <100.0%> (ø)`
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 635f3e7...61c91ee. Read the comment docs.

# Conflicts: # docs/source/release_notes.rst

gsheni · 2021-08-31T18:54:30Z

evalml/pipelines/components/transformers/preprocessing/transform_primitive_components.py

        features.set_index(X_ww.index, inplace=True)

        X_ww = X_ww.ww.drop(self._columns)
+        features.ww.init(logical_types={col_: "categorical" for col_ in features})


So you're taking all features generated from DFS and Calculate Feature Matrix, and making them categorical?

Yes but currently this only applies for URL and Email features. It was the old behavior but now the problem is that ww 0.6.1 infers features created by ft as Unknown.

Yes, after discussion with @dsherry and @freddyaboulton this is what we decided on to address the test issues for this transformer

freddyaboulton

Thank you @ParthivNaresh ! I know it must have been tough to update all of the tests. I agree with the general approach of modifying the set-up code/test fixtures to recover what the results of the inference would have been in 0.5.1.

Nothing "dangerous" stood out to me hehe but I left some questions about some of the modifications we've made here. I look forward to what comes out of #2715 . Hopefully we can cut out ww type inference from our unit tests for greater stability in the future!

evalml/model_understanding/permutation_importance.py

evalml/tests/component_tests/test_delayed_features_transformer.py

freddyaboulton · 2021-08-31T18:11:49Z

evalml/tests/component_tests/test_imputer.py

        {
            "categorical col": pd.Series(
-                ["zero", "one", "two", "zero", "three"], dtype="category"
+                ["zero", "one", "two", "zero", "two"] * 4, dtype="category"


So to make sure I understand, this will now get inferred as categorical because it has three unique values out of 20 values, so n_unique/total_values = 0.15 which is below 0.20?

Sounds good to me. Although I'm worried what will happen if ww ever changes the threshold again. I'm good to proceed with this approach! I would be interested in trying to cut out the inference from our tests entirely - Maybe #2715 is intended to track that @chukarsten ?

evalml/tests/component_tests/test_lgbm_classifier.py

evalml/tests/conftest.py

evalml/tests/data_checks_tests/test_invalid_targets_data_check.py

evalml/tests/model_understanding_tests/prediction_explanations_tests/test_force_plots.py

freddyaboulton · 2021-08-31T19:02:19Z

evalml/pipelines/components/transformers/preprocessing/transform_primitive_components.py

        features.set_index(X_ww.index, inplace=True)

        X_ww = X_ww.ww.drop(self._columns)
+        features.ww.init(logical_types={col_: "categorical" for col_ in features})


Yes but currently this only applies for URL and Email features. It was the old behavior but now the problem is that ww 0.6.1 infers features created by ft as Unknown.

thehomebrewnerd · 2021-08-31T19:59:15Z

@freddyaboulton I can't reply to your comment about the categorical threshold directly for some reason, but the categorical inference threshold is a configurable setting in Woodwork, so if you wanted to use a different value in your tests (or in EvalML overall) you can do so.

You can also set this inside a context manager if you wanted to do it temporarily in a test. Here is an example which shows how you would do that (using a different config option though).

freddyaboulton · 2021-08-31T20:11:55Z

Thank you @thehomebrewnerd ! @ParthivNaresh did you try this? I think we talked about it at the beginning but not sure what came of it. It seems like setting categorical_threshold to 0 and numeric_categorical_threshold to None would recover the old behavior?

ParthivNaresh · 2021-08-31T20:15:14Z

@freddyaboulton @thehomebrewnerd Is this something we want to do? While this did require a decent number of changes, getting rid of the sampling inference + setting a threshold seems like a good long term move. Wouldn't setting categorical_threshold to 0 prevent anything from being recognized as Categorical?

freddyaboulton · 2021-08-31T20:25:19Z

@ParthivNaresh Ah my bad, maybe I meant 1? I'm ok with the approach we have here! But my concern is that if the default threshold value ever changes in the future, we'll have to modify the test cases again to increase/decrease the length so that they match the new threshold. I think setting the thresholds in the tests would prevent us from having to do that? I'm down to kick that out into #2715 though. @chukarsten Maybe we should we rename that epic "Minimize impact of default ww inference in unit tests"?

thehomebrewnerd · 2021-08-31T20:25:55Z

Wouldn't setting categorical_threshold to 0 prevent anything from being recognized as Categorical?

@ParthivNaresh Yes. If you want a series to be recognized as Categorical "more easily" you would want to increase the threshold.

chukarsten · 2021-08-31T15:50:12Z

evalml/model_understanding/permutation_importance.py

        X_permuted.iloc[:, col_idx] = col
+        X_permuted.ww.init(schema=X_features.ww.schema)


Is the schema being invalidated by setting the col? Should we file a WW issue for this?

Modifying the dataframe outside of Woodwork, as is being done with X_permuted.iloc[:, col_idx] = col, always carries the risk of invalidating the schema.

That said, I don't think it is currently possible to do this type of assignment through WW currently with this:

X_permuted.ww.iloc[:, col_idx] = col

I'm not sure how easy it would be to implement, but we could look into it if needed.

By the way, based on my limited testing, as long as the new values don't change the column dtype, the schema should not be invalidated by this type of assignment outside of WW.

evalml/tests/component_tests/test_imputer.py

chukarsten · 2021-08-31T16:14:14Z

evalml/tests/component_tests/test_imputer.py

+            "bool col": [True, False, False, True, True] * 4,
            "categorical with nan": pd.Series(
-                ["0", "1", "0", "0", "3"], dtype="category"
+                ["0", "1", "0", "0", "3"] * 4, dtype="category"


I get that our tests are pretty synthetic, but is it worth pushing back on Woodwork here? It seems like these columns should very much be inferred as categorical, despite being small.

FYI, if you are setting the dtype of the column explicitly to category, Woodwork should recognize this as Categorical, regardless of how many values are present.

An example:

>>> import pandas as pd >>> import woodwork as ww >>> df = pd.DataFrame({'id': [0], 'vals': pd.Series(['1'], dtype='category')}) >>> df.ww.init() >>> df.ww Physical Type Logical Type Semantic Tag(s) Column id int64 Integer ['numeric'] vals category Categorical ['category']

So, that's another option instead of changing series length or updating the inference threshold.

evalml/tests/component_tests/test_imputer.py

chukarsten · 2021-08-31T16:25:46Z

evalml/tests/component_tests/test_lgbm_regressor.py

    X2_fit = pd.DataFrame({"feature": ["c", "b", "a", "d"]})
    X2_predict = pd.DataFrame({"feature": ["d", "c", "b", "a"]})
    X2_predict_expected = pd.DataFrame({0: [3.0, 2.0, 1.0, 0.0]}, dtype="category")
+    X2_fit.ww.init(logical_types={"feature": "categorical"})


This feels like we're having to work around inference rather than using it...

evalml/tests/component_tests/test_one_hot_encoder.py

evalml/tests/component_tests/test_target_encoder.py

chukarsten · 2021-08-31T19:45:46Z

evalml/tests/component_tests/test_imputer.py

        {
            "categorical col": pd.Series(
-                ["zero", "one", "two", "zero", "three"], dtype="category"
+                ["zero", "one", "two", "zero", "two"] * 4, dtype="category"


That was the idea @freddyaboulton . The intent is to try and move as many of these dataframes into a common location, at least within the testing module so that we can avoid the more laborious upgrade lifts. Like this lol

evalml/tests/component_tests/test_lgbm_classifier.py

…valml into woodwork-upgrade-0.6.0

release notes

03e129d

Upgrade woodwork versions

0dac5a0

gsheni requested a review from jeff-hernandez August 24, 2021 20:24

gsheni assigned ParthivNaresh Aug 24, 2021

gsheni removed the request for review from jeff-hernandez August 24, 2021 20:24

ParthivNaresh added 24 commits August 25, 2021 13:17

extend length of columns to identify as categorical

cdcea2c

Merge branch 'main' into woodwork-upgrade-0.6.0

8ef3541

# Conflicts: # docs/source/release_notes.rst

data checks updated

ab4d80e

more component tests

d2d045e

lgbm updates

7f1f92a

no message

eba1d4b

Merge branch 'main' into woodwork-upgrade-0.6.0

9359556

# Conflicts: # docs/source/release_notes.rst

model understanding updates

22c79c1

imputer fixes

7db5787

Merge branch 'main' into woodwork-upgrade-0.6.0

df4edc7

Merge branch 'main' into woodwork-upgrade-0.6.0

f3d4279

test target encoder and invalid target data checks

240655d

one hot encoder updates

708ea7b

more ohe

63eb23a

segmentation fault

86ca452

lgbm, per column, simple imputer

7871c74

imputer and partial dependence

c336f9d

pip install scikit-learn

1fd6231

install woodwork

b086a16

no message

370f337

test_explainers

64868af

plotly update

6b33064

partial dependence

d56a5ca

lint fixes

77374a9

ParthivNaresh requested review from angela97lin, bchen1116, christopherbunn, chukarsten, dsherry, eccabay, freddyaboulton and jeremyliweishih and removed request for freddyaboulton August 31, 2021 15:46

gsheni reviewed Aug 31, 2021

View reviewed changes

freddyaboulton approved these changes Aug 31, 2021

View reviewed changes

test updates

a562c67

chukarsten mentioned this pull request Aug 31, 2021

Merge DF/Series into Fixture : test_imputer.py #2718

Open

chukarsten mentioned this pull request Aug 31, 2021

Merge DF/Series into Fixture : test_component_graph.py #2722

Open

more updates

482a0d6

chukarsten approved these changes Aug 31, 2021

View reviewed changes

chukarsten and others added 4 commits August 31, 2021 16:52

Merge branch 'main' into woodwork-upgrade-0.6.0

e0ddd56

featuretools upgrade

3a353ea

Merge branch 'woodwork-upgrade-0.6.0' of https://github.com/alteryx/e…

209a020

…valml into woodwork-upgrade-0.6.0

lint fix

61c91ee

ParthivNaresh merged commit 6607edf into main Aug 31, 2021

freddyaboulton mentioned this pull request Aug 31, 2021

Automated Latest Dependency Updates #2723

Closed

chukarsten mentioned this pull request Sep 1, 2021

Release v0.32.0 #2729

Merged

freddyaboulton deleted the woodwork-upgrade-0.6.0 branch May 13, 2022 15:02

		X_permuted.iloc[:, col_idx] = col
		X_permuted.ww.init(schema=X_features.ww.schema)

Upgrade Woodwork to 0.6.0 #2690

Upgrade Woodwork to 0.6.0 #2690

Uh oh!

Conversation

ParthivNaresh commented Aug 24, 2021

Uh oh!

codecov bot commented Aug 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehomebrewnerd commented Aug 31, 2021

Uh oh!

freddyaboulton commented Aug 31, 2021

Uh oh!

ParthivNaresh commented Aug 31, 2021

Uh oh!

freddyaboulton commented Aug 31, 2021

Uh oh!

thehomebrewnerd commented Aug 31, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thehomebrewnerd Aug 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Aug 24, 2021 •

edited

Loading

thehomebrewnerd Aug 31, 2021 •

edited

Loading