Update data check actions API to return options instead of actions and add functionality to suggest and take action on columns with null values #3182

angela97lin · 2022-01-05T05:27:58Z

Closes #3116, closes #3013, closes #3144, closes #3243, closes #3240

Everything in this branch should have been previously reviewed; I just didn't want to continuously break and fix any code that was using the intermediate API changes as I broke down issues into smaller (and hopefully easier to review) PRs.

* init * init * start updating tests * add validation code for option * remove data check updates for now * start to clean up tests and add validate_parameter tests * revert highly null dc * add in more valueerror test checking * fix test and logic for column parameters * init * update some tests to new API * fix more tests * fix doc and sparsity test * fix integration tests * fix doctests * fix doctests * fix ts splitting test and impl * fix more ts splitting tests * fix doctest for ts data check * release notes * update target leakage data check docstring for consistency * doctest linting and cleanup * linting

codecov · 2022-01-05T05:32:33Z

Codecov Report

Merging #3182 (81288de) into main (2cb356d) will increase coverage by 0.1%.
The diff coverage is 99.9%.

@@           Coverage Diff           @@
##            main   #3182     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        326     326             
  Lines      31563   31698    +135     
=======================================
+ Hits       31472   31607    +135     
  Misses        91      91

Impacted Files	Coverage Δ
evalml/data_checks/data_check.py	`100.0% <ø> (ø)`
...tests/data_checks_tests/test_data_check_message.py	`100.0% <ø> (ø)`
..._tests/test_data_checks_and_actions_integration.py	`99.0% <98.0%> (-1.0%)`	⬇️
evalml/automl/automl_search.py	`99.9% <100.0%> (+0.1%)`	⬆️
evalml/data_checks/__init__.py	`100.0% <100.0%> (ø)`
evalml/data_checks/class_imbalance_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/data_check_action_option.py	`100.0% <100.0%> (ø)`
evalml/data_checks/data_check_message.py	`100.0% <100.0%> (ø)`
evalml/data_checks/data_check_message_code.py	`100.0% <100.0%> (ø)`
evalml/data_checks/data_checks.py	`100.0% <100.0%> (ø)`
... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2cb356d...81288de. Read the comment docs.

…taCheckAction` (#3152) * init * init * start updating tests * add validation code for option * remove data check updates for now * start to clean up tests and add validate_parameter tests * revert highly null dc * add in more valueerror test checking * fix test and logic for column parameters * init * update some tests to new API * fix more tests * fix doc and sparsity test * fix integration tests * fix doctests * init, update no variance * update highly null dc * fix id column dc * update target leakage dc * update sparsity dc * outliers and uniqueness dc * update target dis dc * update invalid target data check * fix doctests * fix ts splitting test and impl * fix dc validate and tests * fix more ts splitting tests * fix data check actions notebook * update to remove columns to drop * freeze * remove rows to drop * update dc tests * update doctests * fix integration tests * revert requirements * update parameters to set to empty dict * fix tests * fix doctests * retrigger * revert parameter for data check option * fix data check option test * fix more tests from updating default parameters * release notes * use empty instead of none * release notes * cleanup unnecessary code * move logic to data check option class and rename * update integration tests * add tests and some cleanup * fix tests * fix pipeline util test * remove unnecessary conditional * fix naming, need to fix tests * fix tests * fix doctest * add new enums * add logic for enums * update files to use enum * add in testing for invalid enum * fix doctest by updating to_dict impl * linting * try with different base * oops revert yaml * fix tests * remove outdated code * add more tests

… to return impute action for non-highly null columns. (#3197) * init * init * start updating tests * add validation code for option * remove data check updates for now * start to clean up tests and add validate_parameter tests * revert highly null dc * add in more valueerror test checking * fix test and logic for column parameters * init * update some tests to new API * fix more tests * fix doc and sparsity test * fix integration tests * fix doctests * init, update no variance * update highly null dc * fix id column dc * update target leakage dc * update sparsity dc * outliers and uniqueness dc * update target dis dc * update invalid target data check * fix doctests * fix ts splitting test and impl * fix dc validate and tests * fix more ts splitting tests * fix data check actions notebook * update to remove columns to drop * freeze * remove rows to drop * update dc tests * update doctests * fix integration tests * revert requirements * update parameters to set to empty dict * fix tests * fix doctests * retrigger * revert parameter for data check option * fix data check option test * fix more tests from updating default parameters * release notes * use empty instead of none * release notes * cleanup unnecessary code * move logic to data check option class and rename * init rename * retrigger * revert release nots * add new logic for detecting null cols * update integration tests * add tests and some cleanup * fix tests * fix pipeline util test * remove unnecessary conditional * fix test and iteration for null data check * fix naming, need to fix tests * fix tests * fix doctest * add new enums * add logic for enums * update files to use enum * add in testing for invalid enum * fix doctest by updating to_dict impl * linting * try with different base * oops revert yaml * fix tests * remove outdated code * oops fix merge * add more tests * fix null data check tests * update to use per column strategy and fix tests * fix tests for data checks * fix tests and doctests * release notes * fix release notes * oops update action code * oops fix test * update wording of messages * move logic out of dict * update mode to most_frequent for impute strategies * oops fix linting and doctests

* init * more cleanup * begin to clean up tests * fix more tests * updating naming to action_options and fixing more tests * fix no variance * fix another test * fixing automl tests * oops actually fix automl tests * fix the other automl tests * fixing notebook * fix data check tests * fix doctests and docs * integration tests and cleanup * cleanup based on comments * linting * clean up notebook and tests

…3237) * init * init * start updating tests * add validation code for option * remove data check updates for now * start to clean up tests and add validate_parameter tests * revert highly null dc * add in more valueerror test checking * fix test and logic for column parameters * init * update some tests to new API * fix more tests * fix doc and sparsity test * fix integration tests * fix doctests * init, update no variance * update highly null dc * fix id column dc * update target leakage dc * update sparsity dc * outliers and uniqueness dc * update target dis dc * update invalid target data check * fix doctests * fix ts splitting test and impl * fix dc validate and tests * fix more ts splitting tests * fix data check actions notebook * update to remove columns to drop * freeze * remove rows to drop * update dc tests * update doctests * fix integration tests * revert requirements * update parameters to set to empty dict * fix tests * fix doctests * retrigger * revert parameter for data check option * fix data check option test * fix more tests from updating default parameters * release notes * use empty instead of none * release notes * cleanup unnecessary code * move logic to data check option class and rename * init rename * retrigger * revert release nots * add new logic for detecting null cols * update integration tests * add tests and some cleanup * fix tests * fix pipeline util test * remove unnecessary conditional * fix test and iteration for null data check * fix naming, need to fix tests * fix tests * fix doctest * add new enums * add logic for enums * update files to use enum * add in testing for invalid enum * fix doctest by updating to_dict impl * linting * try with different base * oops revert yaml * fix tests * remove outdated code * oops fix merge * add more tests * fix null data check tests * update to use per column strategy and fix tests * fix tests for data checks * fix tests and doctests * release notes * fix release notes * init * init and fix testing * fix integration test * updates test * clean up docs * lint notebook * fix tests with types * linting * release notes * update wording * update impl for natural language and datetimes, remove old tests * fix tests * fix doctest

…lml into update_data_check_action_API

chukarsten

This looks good, Angela! I'm sorry I wasn't able to post some of the reviews I did of your constituent PRs, I had a few pending submission when they got merged. But they were all approvals anyways. This looks good to me and is great work. Way to really stick it through...this one must have been exhausting.

chukarsten · 2022-01-20T17:45:34Z

evalml/tests/data_checks_tests/test_data_check_action_option.py

+        parameters={
+            "impute_strategies": {
+                "parameter_type": "column",
+                "columns": {
+                    "some_column": {
+                        "impute_strategy": {
+                            "categories": ["mean", "most_frequent"],
+                            "type": "category",
+                            "default_value": "most_frequent",
+                        },
+                        "fill_value": {"type": "float", "default_value": 0.0},
+                    },
+                    "some_other_column": {
+                        "impute_strategy": {
+                            "categories": ["mean", "most_frequent"],
+                            "type": "category",
+                            "default_value": "mean",
+                        },
+                        "fill_value": {"type": "float", "default_value": 1.0},
+                    },
+                },
+            }
+        },
+    )


freddyaboulton

@angela97lin This looks good to me! Thanks for all the work in this refactor.

I skimmed this PR as I reviewed all the sub PRs. Only one thing stood out to me.

freddyaboulton · 2022-01-21T20:34:58Z

evalml/data_checks/null_data_check.py

+            X, pct_null_col_threshold=self.pct_null_col_threshold
+        )
+
+        X_to_check_for_any_null = X.ww.select(


I think we can improve this implementation. Right now we do two scans of the data to determine the highly null columns and the columns with any nulls. Doing so also involves creating a new copy of the data without NaturalLanguage/DateTime.

I think if we refactor get_null_column_information to return {col_name: % nulls} we only have to do one scan without creating another copy to determine which are highly null and which are null.

Let's file an issue to do that in a separate PR.

You're right, thank you for pointing out that nuance! I filed #3273 to track this :)

ParthivNaresh

Fantastic work on this, I can't wait for it to be merged. And great coordination on splitting it up into manageable PRs first!

eccabay

Fantastic! Just left a few nits

docs/source/release_notes.rst

eccabay · 2022-01-24T18:45:35Z

docs/source/release_notes.rst

@@ -38,7 +48,6 @@
        * Changed the default objective to ``MedianAE`` from ``R2`` for time series regression :pr:`3205`
        * Removed all-nan Unknown to Double logical conversion in ``infer_feature_types`` :pr:`3196`
        * Checking the validity of holdout data for time series problems can be performed by calling ``pipelines.utils.validate_holdout_datasets`` prior to calling ``predict`` :pr:`3208`
-        * Uncapped numba version and removed it from requirements :pr:`3263`


Should this have moved?

@eccabay Haha yes! I put it in the wrong place before and am sneaking this in here :)

docs/source/release_notes.rst

eccabay · 2022-01-24T18:57:02Z

evalml/data_checks/class_imbalance_data_check.py

+

Another extra space?

eccabay · 2022-01-24T19:09:35Z

evalml/pipelines/utils.py

+                impute_strategies = parameters["impute_strategies"]
+                components.append(
+                    PerColumnImputer(
+                        impute_strategies=impute_strategies, impute_all=False


With the merge of #3267, we don't need the impute_all parameter any more!

That's true! Thank you @eccabay 🙏

angela97lin added 8 commits January 7, 2022 22:03

Merge branch 'main' into update_data_check_action_API

bace330

Merge branch 'main' into update_data_check_action_API

0614cf4

Merge branch 'main' into update_data_check_action_API

c8d9eb2

fix merging main issues

e1c373f

Merge branch 'main' into update_data_check_action_API

6558c9d

Merge branch 'main' into update_data_check_action_API

52de764

Merge branch 'main' into update_data_check_action_API

b48fa97

angela97lin changed the title ~~Update data check actions API~~ Update data check actions API to return options instead of actions and add functionality to suggest and take action on columns with null values Jan 12, 2022

angela97lin added 2 commits January 13, 2022 01:22

Merge branch 'main' into update_data_check_action_API

c98c96e

angela97lin mentioned this pull request Jan 13, 2022

NaturalLanguageDataCheck should raise an error for highly null columns and a warning for columns with nulls #3240

Closed

angela97lin added 3 commits January 13, 2022 13:53

Merge branch 'main' into update_data_check_action_API

4f6952a

Merge branch 'main' into update_data_check_action_API

ad9eacb

Merge branch 'main' into update_data_check_action_API

b24c792

angela97lin marked this pull request as ready for review January 13, 2022 21:01

auto-assign bot assigned angela97lin Jan 13, 2022

angela97lin requested review from ParthivNaresh, chukarsten, freddyaboulton, bchen1116, dsherry, eccabay and jeremyliweishih January 13, 2022 21:01

angela97lin marked this pull request as draft January 13, 2022 21:02

angela97lin removed request for dsherry, jeremyliweishih and bchen1116 January 13, 2022 21:02

angela97lin added 5 commits January 18, 2022 14:05

merging

888d901

release notes

f993939

Merge branch 'update_data_check_action_API' of github.com:alteryx/eva…

8037346

…lml into update_data_check_action_API

angela97lin marked this pull request as ready for review January 19, 2022 05:42

angela97lin requested review from ParthivNaresh, chukarsten, freddyaboulton, bchen1116, christopherbunn, dsherry, eccabay and jeremyliweishih January 19, 2022 05:43

minor cleanup

5208a1a

chukarsten approved these changes Jan 20, 2022

View reviewed changes

angela97lin mentioned this pull request Jan 20, 2022

Removes DateTimeNaNDataCheck and NaturalLanguageNaNDataCheck #3260

Merged

release notes

6249d74

freddyaboulton approved these changes Jan 21, 2022

View reviewed changes

angela97lin mentioned this pull request Jan 22, 2022

Improve implementation of NullDataCheck #3273

Open

ParthivNaresh approved these changes Jan 24, 2022

View reviewed changes

Merge branch 'main' into update_data_check_action_API

5276a69

eccabay approved these changes Jan 24, 2022

View reviewed changes

angela97lin added 4 commits January 24, 2022 17:48

Merge branch 'main' into update_data_check_action_API

1094e7b

update release notes

4e8062b

remove impute all

aefaf14

linting

81288de

angela97lin merged commit 9e36684 into main Jan 25, 2022

angela97lin deleted the update_data_check_action_API branch January 25, 2022 17:50

angela97lin mentioned this pull request Jan 25, 2022

Release 0.43.0 #3279

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update data check actions API to return options instead of actions and add functionality to suggest and take action on columns with null values #3182

Update data check actions API to return options instead of actions and add functionality to suggest and take action on columns with null values #3182

angela97lin commented Jan 5, 2022 •

edited

Loading

codecov bot commented Jan 5, 2022 •

edited

Loading

chukarsten left a comment

chukarsten Jan 20, 2022

freddyaboulton left a comment

freddyaboulton Jan 21, 2022

angela97lin Jan 22, 2022

ParthivNaresh left a comment

eccabay left a comment

eccabay Jan 24, 2022

angela97lin Jan 25, 2022

eccabay Jan 24, 2022

eccabay Jan 24, 2022

angela97lin Jan 25, 2022

Update data check actions API to return options instead of actions and add functionality to suggest and take action on columns with null values #3182

Update data check actions API to return options instead of actions and add functionality to suggest and take action on columns with null values #3182

Conversation

angela97lin commented Jan 5, 2022 • edited Loading

codecov bot commented Jan 5, 2022 • edited Loading

Codecov Report

chukarsten left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ParthivNaresh left a comment

Choose a reason for hiding this comment

eccabay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin commented Jan 5, 2022 •

edited

Loading

codecov bot commented Jan 5, 2022 •

edited

Loading