Make EvalML compatible with Woodwork changes #4066

ParthivNaresh · 2023-03-09T20:21:56Z

This PR updates EvalML tests to be compatible with Woodwork's new numeric logical type inference for incoming object dtypes and dependence calculations that treats booleans as numeric.

codecov · 2023-03-09T20:34:25Z

Codecov Report

Merging #4066 (632b124) into main (f509df4) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #4066     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        349     349             
  Lines      37509   37514      +5     
=======================================
+ Hits       37391   37396      +5     
  Misses       118     118

Impacted Files	Coverage Δ
...omponents/estimators/regressors/arima_regressor.py	`100.0% <ø> (ø)`
...tors/regressors/exponential_smoothing_regressor.py	`100.0% <ø> (ø)`
...ponents/estimators/regressors/prophet_regressor.py	`100.0% <ø> (ø)`
...ponents/estimators/regressors/xgboost_regressor.py	`100.0% <ø> (ø)`
...mponent_tests/test_column_selector_transformers.py	`100.0% <ø> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`99.9% <ø> (ø)`
evalml/pipelines/classification_pipeline.py	`100.0% <100.0%> (ø)`
...valml/pipelines/components/estimators/estimator.py	`100.0% <100.0%> (ø)`
...s/components/estimators/regressors/et_regressor.py	`100.0% <100.0%> (ø)`
...s/components/estimators/regressors/rf_regressor.py	`100.0% <100.0%> (ø)`
... and 11 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

…m/alteryx/evalml into make_compat_with_object_inference

…make_compat_with_object_inference

Reformat doc

docs/source/release_notes.rst

bchen1116

Nice!

…m/alteryx/evalml into make_compat_with_object_inference

jeremyliweishih

LGTM pending prediction explanations discussion

jeremyliweishih · 2023-03-14T17:22:44Z

docs/source/user_guide/timeseries.ipynb

@@ -131,6 +131,7 @@
   "outputs": [],
   "source": [
    "X[\"Categorical\"] = [str(i % 4) for i in range(len(X))]\n",
+    "X[\"Categorical\"] = X[\"Categorical\"].astype(\"category\")\n",


do you want to add a comment here and below explaining why we're setting it as category instead of leaving it as object? Might be good to remember.

jeremyliweishih · 2023-03-14T17:24:56Z

evalml/tests/model_understanding_tests/prediction_explanations_tests/test_explainers.py

@@ -1148,7 +1148,20 @@ def test_json_serialization(
        num_to_explain=1,
        output_format="dict",
    )
-    assert json.loads(json.dumps(best_worst)) == best_worst
+
+    def str_to_num_decoder(dct):


can you explain why we need this now? do we need to add support directly into prediction explanations?

I'm concerned by this change as well. I think json.loads(json.dumps(best_worst)) == best_worst is an axiom we should maintain, rather than patching it over for the sake of the upgrade. What change predicated this, and how can we account for it in prediction explanations instead?

This is because the output of explain_predictions_best_worst results in the keys of predicted_values being integers, since they're numeric values being represented as strings. Woodwork's new inference will infer strings as their appropriate numeric logical type if their incoming dtype is object and no other logical type is specified.

'predicted_values': {'error_name': 'Cross Entropy', 'error_value': 9.967001210947085e-07, 'index_id': 33, 'predicted_value': 1, 'probabilities': {0: 0.0, 1: 1.0}, 'target_value': 1},

but JSON doesn't allow non-string keys, so it replaces them with

'predicted_values': {'error_name': 'Cross Entropy', 'error_value': 9.967001210947085e-07, 'index_id': 33, 'predicted_value': 1, 'probabilities': {'0': 0.0, '1': 1.0}, # String keys 'target_value': 1},

What I added here was an object_hook to be called instead of the standard dict which converts the string key back to an int just so they can be compared for the test.

I think what's happening is that this is creating keys and assigning them to the probabilities key even if they're integers. I can modify this to stringify all keys since JSON will end up converting them anyways. I just thought it would be better for the output of explain_predictions_best_worst to represent the actual values even if JSON would end up stringifying them. @eccabay @jeremyliweishih does that make sense?

wait @ParthivNaresh why does woodwork inference matter when we're testing if dumping and loading the same json will result in the same dictionary?

@jeremyliweishih Because JSON only allows key names to be strings, so if the values themselves aren't strings, then they get dumped as integers and read as strings. I think somewhere in the prediction explanation process, the original dtype of y is lost and reinferred, which means if the original dtype was a string, it would be reinferred as an integer.

I tried replacing

elif problem_type == problem_type.BINARY: X, y = X_y_binary y = pd.Series(y).astype("str") pipeline = logistic_regression_binary_pipeline

with

elif problem_type == problem_type.BINARY: X, y = X_y_binary y = ww.init_series(pd.Series(y), logical_type="Unknown") pipeline = logistic_regression_binary_pipeline

or explicitly setting it as string dtype (string instead of str), but it still writes out the dictionary in best_worst as an integer

I just thought it would be better for the output of explain_predictions_best_worst to represent the actual values even if JSON would end up stringifying them

Doesn't your second comment contradict this, so explain_predictions_best_worst is still modifying the actual values, but in the wrong direction?

My preference remains that we keep the assertion that json.loads(json.dumps(best_worst)) == best_worst, and we update the code to maintain this rather than the test.

explain_predictions_best_worst is modifying this somewhere but that's not the expectation. If I pass in a string dtype (which Woodwork won't infer as numeric, but Categorical in this case) then that would be the expected output.

I'm taking a look into explain_predictions_best_worst to see where this unexpected reinference is happening

@eccabay I think the issue was that ww.init_series was being used to determine the unique classes that existed in a classification pipeline. If the column names were strings then they got inferred as integers because of this.

docs/source/release_notes.rst

eccabay · 2023-03-14T17:16:05Z

evalml/tests/data_checks_tests/test_id_columns_data_check.py

+    if input_type == "string":
+        order = ["col_2", "col_3_id", "col_1_id"]
+        order_msg = "Columns 'col_2', 'col_3_id', 'col_1_id' are 95.0% or more likely to be an ID column"
+    else:
+        order = ["col_2", "col_1_id", "col_3_id"]
+        order_msg = "Columns 'col_2', 'col_1_id', 'col_3_id' are 95.0% or more likely to be an ID column"
+


What change caused this? It's a surprising distinction to make

Also, a nitpicky clarity change - we can replace the separate order messages here with order_msg = f"Columns '{order[0]}', '{order[1]}', '{order[2]}' are 95% or more..." just to make it clearer what the difference is!

In this case, col_3_id is a random series of string-represented numbers, which under the new inference will become the Integer ltype. Following this logic in validate, since they're all unique values and since the column ends in id, it gets identified as a possible ID column.

Previously however, since this column wasn't a numeric logical type, the logic here didn't catch it, but this did.

This just ends up resulting in a different ordering by which these columns are caught as possible ID columns by the data check

eccabay · 2023-03-14T17:27:44Z

evalml/tests/model_understanding_tests/prediction_explanations_tests/test_explainers.py

@@ -1148,7 +1148,20 @@ def test_json_serialization(
        num_to_explain=1,
        output_format="dict",
    )
-    assert json.loads(json.dumps(best_worst)) == best_worst
+
+    def str_to_num_decoder(dct):


I'm concerned by this change as well. I think json.loads(json.dumps(best_worst)) == best_worst is an axiom we should maintain, rather than patching it over for the sake of the upgrade. What change predicated this, and how can we account for it in prediction explanations instead?

jeremyliweishih

let's figure out this prediction explanation shenanigans first

bchen1116 and others added 5 commits March 9, 2023 14:15

update file

4e76e5d

update release notes

4ed1c64

lint

0889ad3

update latest dep

080a4cc

update min

dc20f23

bchen1116 and others added 10 commits March 10, 2023 12:03

update test

0141795

Merge branch 'main' into bc_target_leakage_boolean

5996b78

revert woodwork versions

06b87dd

Merge branch 'main' into make_compat_with_object_inference

e2d9401

set ww min to 0.22.0

02f5f7f

Merge branch 'make_compat_with_object_inference' of https://github.co…

a7f8ef2

…m/alteryx/evalml into make_compat_with_object_inference

empty

a9c365c

Merge remote-tracking branch 'origin/bc_target_leakage_boolean' into …

818c838

…make_compat_with_object_inference

merge target leakage update

7ecf770

update wodwork for meta.yaml

d0803e2

ParthivNaresh changed the title ~~[DO NOT MERGE] Make EvalML compatible with new numeric object dtype inference~~ [DO NOT MERGE] Make EvalML compatible with Woodwork changes Mar 14, 2023

remove print

9a0977c

bchen1116 mentioned this pull request Mar 14, 2023

DRAFT: Update targetLeakageDataCheck to handle boolean values #4061

Closed

ParthivNaresh added 2 commits March 14, 2023 11:16

lint

92b979d

Update timeseries.ipynb

c17717e

Reformat doc

ParthivNaresh changed the title ~~[DO NOT MERGE] Make EvalML compatible with Woodwork changes~~ Make EvalML compatible with Woodwork changes Mar 14, 2023

ParthivNaresh requested review from eccabay, jeremyliweishih and bchen1116 and removed request for eccabay, jeremyliweishih and bchen1116 March 14, 2023 15:48

ParthivNaresh requested a review from bchen1116 March 14, 2023 15:49

ParthivNaresh marked this pull request as ready for review March 14, 2023 15:49

auto-assign bot assigned ParthivNaresh Mar 14, 2023

ParthivNaresh requested review from fjlanasa and christopherbunn March 14, 2023 15:49

bchen1116 reviewed Mar 14, 2023

View reviewed changes

docs/source/release_notes.rst Outdated Show resolved Hide resolved

bchen1116 approved these changes Mar 14, 2023

View reviewed changes

ParthivNaresh added 3 commits March 14, 2023 12:25

update release notes

6c8e2b5

Merge branch 'make_compat_with_object_inference' of https://github.co…

032e1dd

…m/alteryx/evalml into make_compat_with_object_inference

retrigger codecov

f0b820c

jeremyliweishih approved these changes Mar 14, 2023

View reviewed changes

eccabay reviewed Mar 14, 2023

View reviewed changes

jeremyliweishih requested changes Mar 14, 2023

View reviewed changes

ParthivNaresh added 2 commits March 14, 2023 16:15

lint

623c4d2

trigger

632b124

jeremyliweishih approved these changes Mar 14, 2023

View reviewed changes

eccabay approved these changes Mar 14, 2023

View reviewed changes

ParthivNaresh enabled auto-merge (squash) March 14, 2023 21:08

ParthivNaresh merged commit 1412fc3 into main Mar 14, 2023

ParthivNaresh deleted the make_compat_with_object_inference branch March 14, 2023 21:15

eccabay mentioned this pull request Mar 15, 2023

Automated Latest Dependency Updates #4075

Closed

chukarsten mentioned this pull request Mar 15, 2023

Release v0.69.0 #4078

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make EvalML compatible with Woodwork changes #4066

Make EvalML compatible with Woodwork changes #4066

ParthivNaresh commented Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 9, 2023 •

edited

Loading

bchen1116 left a comment

jeremyliweishih left a comment

jeremyliweishih Mar 14, 2023

jeremyliweishih Mar 14, 2023

eccabay Mar 14, 2023

ParthivNaresh Mar 14, 2023

jeremyliweishih Mar 14, 2023

ParthivNaresh Mar 14, 2023

eccabay Mar 14, 2023 •

edited

Loading

ParthivNaresh Mar 14, 2023

ParthivNaresh Mar 14, 2023

eccabay Mar 14, 2023

ParthivNaresh Mar 14, 2023

eccabay Mar 14, 2023

jeremyliweishih left a comment

Make EvalML compatible with Woodwork changes #4066

Make EvalML compatible with Woodwork changes #4066

Conversation

ParthivNaresh commented Mar 9, 2023 • edited Loading

codecov bot commented Mar 9, 2023 • edited Loading

Codecov Report

bchen1116 left a comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eccabay Mar 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

ParthivNaresh commented Mar 9, 2023 •

edited

Loading

codecov bot commented Mar 9, 2023 •

edited

Loading

eccabay Mar 14, 2023 •

edited

Loading