Standardize data check messages by adding default "rows" and "columns" metadata by angela97lin · Pull Request #2869 · alteryx/evalml

angela97lin · 2021-10-01T20:44:06Z

Closes #2792.

Adds a default "rows" and "columns" metadata field to data check messages.
Also updates data checks which previously would output one message per column to consolidate all similar messages into one message.

I still need to update docstring tests, will wait until #2933 is merged to avoid unnecessary changes and conflicts.

codecov · 2021-10-01T20:49:40Z

Codecov Report

Merging #2869 (e9e863b) into main (c941353) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2869     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        302     302             
  Lines      28593   28626     +33     
=======================================
+ Hits       28500   28533     +33     
  Misses        93      93

Impacted Files	Coverage Δ
evalml/data_checks/class_imbalance_data_check.py	`100.0% <ø> (ø)`
evalml/data_checks/datetime_format_data_check.py	`100.0% <ø> (ø)`
evalml/data_checks/datetime_nan_data_check.py	`100.0% <ø> (ø)`
evalml/data_checks/invalid_targets_data_check.py	`100.0% <ø> (ø)`
evalml/data_checks/multicollinearity_data_check.py	`100.0% <ø> (ø)`
...lml/data_checks/natural_language_nan_data_check.py	`100.0% <ø> (ø)`
...alml/data_checks/target_distribution_data_check.py	`100.0% <ø> (ø)`
.../data_checks_tests/test_datetime_nan_data_check.py	`100.0% <ø> (ø)`
...s/data_checks_tests/test_highly_null_data_check.py	`100.0% <ø> (ø)`
...ts/data_checks_tests/test_id_columns_data_check.py	`100.0% <ø> (ø)`
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c941353...e9e863b. Read the comment docs.

…ml into 2792_standardize_dc_cleanup

bchen1116

LGTM! Left some nits, but I like that we have now simplified the returns and combined the column/rows when needed!

evalml/data_checks/no_variance_data_check.py

bchen1116 · 2021-10-15T16:53:02Z

evalml/data_checks/target_leakage_data_check.py

            >>> target_leakage_check = TargetLeakageDataCheck(pct_corr_threshold=0.95)
            >>> assert target_leakage_check.validate(X, y) == {
-            ...     "warnings": [{"message": "Column 'leak' is 95.0% or more correlated with the target",
+            ...     "warnings": [{"message": "Columns 'leak' are 95.0% or more correlated with the target",


It feels weird that this is plural when there's only one column. Not a nit, but would be nice to fix this

…ml into 2792_standardize_dc_cleanup

ParthivNaresh

Looks great, it's definitely odd that the docstrings aren't failing in HighlyNullDataCheck and elsewhere. Getting this compatible with tempo health shouldn't be a heavy lift from here for me, thanks for the changes!

evalml/data_checks/highly_null_data_check.py

ParthivNaresh · 2021-10-18T22:27:47Z

evalml/tests/data_checks_tests/test_target_leakage_data_check.py

-            ).to_dict(),
-            DataCheckWarning(
-                message="Column 'd' is 80.0% or more correlated with the target",
+                message="Columns 'a', 'b', 'c', 'd' are 80.0% or more correlated with the target",


The beauty of consolidation

ParthivNaresh · 2021-10-20T21:25:00Z

evalml/data_checks/outliers_data_check.py

            DataCheckAction(
                DataCheckActionCode.DROP_ROWS,
-                metadata={"indices": all_rows_with_indices},
+                metadata={"rows": all_rows_with_indices},


I guess a follow up comment, do we want to repeat the all_rows_with_indices over here? Maybe it should exist only in the DataCheckAction and not the Warning?

Oof I missed this comment before merging, but I'm pretty indifferent about this either way!

init idea

0ac5f71

angela97lin self-assigned this Oct 1, 2021

angela97lin added 20 commits October 5, 2021 16:27

Merge branch 'main' into 2792_standardize_dc_cleanup

fdb7c38

fix silly update mistake

9d3b6f4

fix data check message tests

41ef37d

mini checkpoint, add to data check action

daba159

minor cleanup, still in process

c6194e0

cleaned up highly null

d98d3f3

update outlier data check and tests

b7960ff

begin to change other data checks, expecting tests to break

c33d7f4

Merge branch 'main' into 2792_standardize_dc_cleanup

40aaba5

a little more cleanup for target leakage and uniqueness

9c86afb

more cleanup

b2bc935

Merge branch 'main' into 2792_standardize_dc_cleanup

a11a010

clean up no variance data check

f9483d7

Merge branch 'main' into 2792_standardize_dc_cleanup

1a6395a

awful cleanup of test data checks

ffac77b

Merge branch '2792_standardize_dc_cleanup' of github.com:alteryx/eval…

961c033

…ml into 2792_standardize_dc_cleanup

fix classification test

ab463b7

merge

1db9839

more fixing of tests

463e33f

linting

e06d9ec

angela97lin changed the title ~~[Spike] Standardize data check messages (columns vs column)~~ Standardize data check messages by adding default "rows" and "columns" metadata Oct 15, 2021

angela97lin marked this pull request as ready for review October 15, 2021 03:40

angela97lin requested review from bchen1116, chukarsten, dsherry, eccabay and freddyaboulton October 15, 2021 03:41

angela97lin requested review from ParthivNaresh and jeremyliweishih October 15, 2021 03:41

bchen1116 approved these changes Oct 15, 2021

View reviewed changes

angela97lin added 7 commits October 18, 2021 10:16

Merge branch 'main' into 2792_standardize_dc_cleanup

72aaa39

update singlar vs plural and added test

61d2269

Merge branch 'main' into 2792_standardize_dc_cleanup

57855cb

refactor no variance data check code and update release notes

7d3f3fd

Merge branch '2792_standardize_dc_cleanup' of github.com:alteryx/eval…

4a789f9

…ml into 2792_standardize_dc_cleanup

fix singular case for test

ca92559

Merge branch 'main' into 2792_standardize_dc_cleanup

9edf665

angela97lin mentioned this pull request Oct 18, 2021

Updating Data Checks to support Data Health #2907

Merged

ParthivNaresh approved these changes Oct 18, 2021

View reviewed changes

angela97lin added 2 commits October 19, 2021 10:57

merging

d255f47

Merge branch 'main' into 2792_standardize_dc_cleanup

dc09896

ParthivNaresh reviewed Oct 20, 2021

View reviewed changes

angela97lin added 2 commits October 21, 2021 11:48

clean up doctests

c7fffe5

Merge branch 'main' into 2792_standardize_dc_cleanup

e9e863b

angela97lin merged commit 21aca40 into main Oct 21, 2021

angela97lin deleted the 2792_standardize_dc_cleanup branch October 21, 2021 16:13

angela97lin mentioned this pull request Oct 26, 2021

Add integration tests for end to end flow for data checks --> data check actions #2883

Merged

chukarsten mentioned this pull request Oct 27, 2021

Release v0.36.0 #2974

Merged

angela97lin mentioned this pull request Nov 16, 2021

Add columns field as metadata that is explicitly enforced in data check output #2814

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardize data check messages by adding default "rows" and "columns" metadata#2869

Standardize data check messages by adding default "rows" and "columns" metadata#2869
angela97lin merged 32 commits intomainfrom
2792_standardize_dc_cleanup

angela97lin commented Oct 1, 2021 •

edited

Loading

Uh oh!

codecov bot commented Oct 1, 2021 •

edited

Loading

Uh oh!

bchen1116 left a comment

Uh oh!

Uh oh!

bchen1116 Oct 15, 2021

Uh oh!

ParthivNaresh left a comment

Uh oh!

Uh oh!

ParthivNaresh Oct 18, 2021

Uh oh!

ParthivNaresh Oct 20, 2021

Uh oh!

angela97lin Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

angela97lin commented Oct 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bchen1116 Oct 15, 2021

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ParthivNaresh Oct 18, 2021

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh Oct 20, 2021

Choose a reason for hiding this comment

Uh oh!

angela97lin Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

angela97lin commented Oct 1, 2021 •

edited

Loading

codecov bot commented Oct 1, 2021 •

edited

Loading