Updating Data Checks to support Data Health #2907

ParthivNaresh · 2021-10-14T05:52:44Z

After conversation with Tyler, Raymond, Raj, and Dylan, we've decided to proceed with Woodwork's box plot for outlier detection, and future work done by Raymond will determine if this is the best approach. If not, the necessary changes can be made in Woodwork's implementation.

# Conflicts: # docs/source/release_notes.rst

codecov · 2021-10-14T13:25:30Z

Codecov Report

Merging #2907 (edd73a4) into main (47c1fec) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2907     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        302     302             
  Lines      28850   28872     +22     
=======================================
+ Hits       28759   28781     +22     
  Misses        91      91

Impacted Files	Coverage Δ
evalml/tests/data_checks_tests/test_data_checks.py	`100.0% <ø> (ø)`
...s/data_checks_tests/test_highly_null_data_check.py	`100.0% <ø> (ø)`
evalml/data_checks/highly_null_data_check.py	`100.0% <100.0%> (ø)`
evalml/data_checks/outliers_data_check.py	`100.0% <100.0%> (ø)`
...ests/data_checks_tests/test_outliers_data_check.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47c1fec...edd73a4. Read the comment docs.

ParthivNaresh · 2021-10-14T13:39:25Z

evalml/data_checks/outliers_data_check.py

        log_scale = (
            -19.8196822259052
-            + 8.5359212447622 * log_n
+            + 18.5359212447622 * log_n


Not sure if we missed this before on purpose but this was part of the original log scale

ParthivNaresh · 2021-10-14T13:40:22Z

evalml/data_checks/outliers_data_check.py

        return results

+    @staticmethod
+    def _get_boxplot_data(data_):


The purpose of this is mainly to provide consumers with easy access to the information it needs without having to instantiate OutliersDataCheck, while still allowing validate to use the numbers it needs so this code doesn't clutter up the function

Maybe we should make it public then?

ParthivNaresh · 2021-10-14T13:42:15Z

evalml/data_checks/outliers_data_check.py

 """Data check that checks if there are any outliers in input data by using IQR to determine score anomalies."""
 import numpy as np
 from scipy.stats import gamma
+from statsmodels.stats.stattools import medcouple


We have to rely on the statsmodels implementation of medcouple because robustats doesn't seem to have a conda installation option. This also limits having to add extra dependencies

ParthivNaresh · 2021-10-14T13:45:26Z

evalml/data_checks/highly_null_data_check.py

-                    details={"pct_null_cols": highly_null_rows},
+                    details={
+                        "pct_null_cols": highly_null_rows,
+                        "pct_of_rows_above_thresh": round(


Necessary for pct_highly_null_rows in table_health

ParthivNaresh · 2021-10-14T13:57:05Z

evalml/data_checks/highly_null_data_check.py

+                    DataCheckActionCode.DROP_COL,
+                    metadata={
+                        "column": col_name,
+                        "row_indices": X[col_name][X[col_name].isnull()].index.tolist(),


Necessary for missing_score in table_health

Correct me if I'm wrong, but looking at the table health code, the missing_score doesn't use the indices to compute the scores. I think this is fine to keep as is while we do the integration but maybe we should file an issue to delete this? I guess I'm worried it can be a very big list.

The score doesn't use this but the indices are part of the payload:

payload = {"score": score, "indices": indices}

and are added to the column_payload in composite_score:

if metric == "missing": missing = missing_score(data) column_scores.append(missing["score"]) column_payload["missing"] = missing

which is then appended to the final payload:

payload["column_scores"].append(column_payload)

evalml/data_checks/outliers_data_check.py

…heck

ParthivNaresh · 2021-10-14T17:01:13Z

evalml/tests/data_checks_tests/test_data_checks.py

                DataCheckActionCode.IMPUTE_COL,
                metadata={"column": None, "is_target": True, "impute_strategy": "mean"},
            ).to_dict(),
+            DataCheckAction(


NoVarianceDataCheck throws this

Maybe we should get @angela97lin 's opinion on this before merging?

I think the intention is to not have duplicate actions for the same column and now there are two DROP_COL for both all_null and also_all_null.

Do we need to include row_indices in the action code? Maybe it can go in the warning?

#2869 ends up combining all of the actions into one action! Agreed though, if the action is just drop the column, the row indices seem unimportant 😛

# Conflicts: # docs/source/release_notes.rst

freddyaboulton

@ParthivNaresh This is great! I think this is pretty much ready to go but I think we should discuss whether we need to include row_indices in the action code for the highly null check. It's introducing a subtle change where there are can now be two DROP_COLs per column.

freddyaboulton · 2021-10-15T17:45:05Z

evalml/data_checks/highly_null_data_check.py

+                    DataCheckActionCode.DROP_COL,
+                    metadata={
+                        "column": col_name,
+                        "row_indices": X[col_name][X[col_name].isnull()].index.tolist(),


Correct me if I'm wrong, but looking at the table health code, the missing_score doesn't use the indices to compute the scores. I think this is fine to keep as is while we do the integration but maybe we should file an issue to delete this? I guess I'm worried it can be a very big list.

freddyaboulton · 2021-10-15T17:58:31Z

evalml/data_checks/outliers_data_check.py

        return results

+    @staticmethod
+    def _get_boxplot_data(data_):


Maybe we should make it public then?

freddyaboulton · 2021-10-15T18:48:07Z

evalml/tests/data_checks_tests/test_data_checks.py

                DataCheckActionCode.IMPUTE_COL,
                metadata={"column": None, "is_target": True, "impute_strategy": "mean"},
            ).to_dict(),
+            DataCheckAction(


Maybe we should get @angela97lin 's opinion on this before merging?

I think the intention is to not have duplicate actions for the same column and now there are two DROP_COL for both all_null and also_all_null.

Do we need to include row_indices in the action code? Maybe it can go in the warning?

docs/source/release_notes.rst

…yx/evalml into Match-DataHealth-Functions

# Conflicts: # evalml/data_checks/highly_null_data_check.py # evalml/tests/data_checks_tests/test_data_checks.py # evalml/tests/data_checks_tests/test_highly_null_data_check.py

freddyaboulton

Looks great to me @ParthivNaresh ! Thanks for making the changes

ParthivNaresh added 7 commits October 3, 2021 12:53

updates

2cb87e7

Merge branch 'main' into Match-DataHealth-Functions

d05bbba

updates

8681b54

release notes

5bb8106

Merge branch 'main' into Match-DataHealth-Functions

af130d2

# Conflicts: # docs/source/release_notes.rst

release notes

456f7d1

release notes

3bbc85b

rely on statsmodels for medcouple

f4137b3

ParthivNaresh commented Oct 14, 2021

View reviewed changes

ParthivNaresh added 2 commits October 14, 2021 10:18

docstring update and import module in function

22f912b

lint fixes

f492f0b

ParthivNaresh commented Oct 14, 2021

View reviewed changes

evalml/data_checks/outliers_data_check.py Outdated Show resolved Hide resolved

update to reflect nulls being counted as values in no variance data c…

1dd7314

…heck

ParthivNaresh commented Oct 14, 2021

View reviewed changes

ParthivNaresh added 2 commits October 14, 2021 15:20

set bounds to woodwork approach

5b056c4

Merge branch 'main' into Match-DataHealth-Functions

6226441

# Conflicts: # docs/source/release_notes.rst

ParthivNaresh marked this pull request as ready for review October 14, 2021 19:48

auto-assign bot assigned ParthivNaresh Oct 14, 2021

Merge branch 'main' into Match-DataHealth-Functions

77aad04

ParthivNaresh requested review from angela97lin, bchen1116, christopherbunn and chukarsten and removed request for angela97lin October 15, 2021 15:10

ParthivNaresh requested review from eccabay, freddyaboulton and jeremyliweishih October 15, 2021 15:10

freddyaboulton suggested changes Oct 15, 2021

View reviewed changes

eccabay reviewed Oct 18, 2021

View reviewed changes

docs/source/release_notes.rst Outdated Show resolved Hide resolved

chukarsten added the priority label Oct 19, 2021

chukarsten changed the title ~~Updating Data Checks to support Tempo Data Health~~ Updating Data Checks to support Data Health Oct 19, 2021

Merge branch 'main' into Match-DataHealth-Functions

a8dd18c

ParthivNaresh marked this pull request as draft October 20, 2021 18:05

ParthivNaresh added 7 commits October 20, 2021 15:37

changes

a908d04

Merge branch 'main' into Match-DataHealth-Functions

c278d53

Merge branch 'Match-DataHealth-Functions' of https://github.com/alter…

a1a3000

…yx/evalml into Match-DataHealth-Functions

Merge branch 'main' into Match-DataHealth-Functions

354784d

# Conflicts: # evalml/data_checks/highly_null_data_check.py # evalml/tests/data_checks_tests/test_data_checks.py # evalml/tests/data_checks_tests/test_highly_null_data_check.py

no message

11edf48

update returned data

7ec0a05

Merge branch 'main' into Match-DataHealth-Functions

8e95d51

ParthivNaresh marked this pull request as ready for review October 21, 2021 20:05

freddyaboulton approved these changes Oct 21, 2021

View reviewed changes

Merge branch 'main' into Match-DataHealth-Functions

edd73a4

ParthivNaresh merged commit 8357f13 into main Oct 22, 2021

chukarsten mentioned this pull request Oct 27, 2021

Release v0.36.0 #2974

Merged

freddyaboulton deleted the Match-DataHealth-Functions branch May 13, 2022 15:23

Updating Data Checks to support Data Health #2907

Updating Data Checks to support Data Health #2907

Uh oh!

Conversation

ParthivNaresh commented Oct 14, 2021 • edited by chukarsten Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Oct 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ParthivNaresh Oct 14, 2021 • edited by chukarsten Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh Oct 14, 2021 • edited by chukarsten Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

freddyaboulton left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ParthivNaresh commented Oct 14, 2021 •

edited by chukarsten

Loading

codecov bot commented Oct 14, 2021 •

edited

Loading

ParthivNaresh Oct 14, 2021 •

edited by chukarsten

Loading

ParthivNaresh Oct 14, 2021 •

edited by chukarsten

Loading

freddyaboulton left a comment •

edited

Loading