Delete data check code from AutoML search #1935

angela97lin · 2021-03-05T18:27:28Z

codecov · 2021-03-08T21:52:58Z

Codecov Report

Merging #1935 (b1b04c1) into main (10297a2) will decrease coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1935     +/-   ##
=========================================
- Coverage   100.0%   100.0%   -0.0%     
=========================================
  Files         273      273             
  Lines       22406    22288    -118     
=========================================
- Hits        22400    22282    -118     
  Misses          6        6

Impacted Files	Coverage Δ
evalml/data_checks/data_checks.py	`100.0% <ø> (ø)`
evalml/tuners/random_search_tuner.py	`95.3% <ø> (ø)`
evalml/automl/automl_search.py	`100.0% <100.0%> (ø)`
evalml/data_checks/__init__.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.0% <100.0%> (ø)`
evalml/tests/data_checks_tests/test_data_checks.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10297a2...b1b04c1. Read the comment docs.

angela97lin · 2021-03-09T16:07:06Z

docs/source/user_guide/data_checks.ipynb

-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### High Variance Cross-Validation Scores\n",


oops, forgot to delete this before!

freddyaboulton

@angela97lin The code changes look great! I think we should synch about whether or not we should delete the data checks section from the automl user guide before merging!

docs/source/release_notes.rst

evalml/automl/automl_search.py

freddyaboulton · 2021-03-09T16:26:57Z

docs/source/user_guide/automl.ipynb

@@ -142,17 +142,6 @@
    "print(regression_objective.name)"
   ]
  },
-  {


I still think we should mention data checks in the automl user guide.

If I'm following the intention of the data check actions epic, we're moving from "AutoML will check your data for you" to "Evalml has functionality to check your data and recommend fixes before you start AutoML".

I think it's important to document the relationship between data checks and automl and that it makes sense to document this relationship here.

So I think we should:

Add back the data checks section (changing it a bit to make it clear that AutoMLSearch is not running the checks)

Import DefaultDataChecks and run on the data.

Say that the results of the data checks don't reveal anything that would cause problems so we can run search.

Link to the data checks user guide page for more information

Let me know what you think!

Hmmm, I like how this sounds... my only qualm about this is it would make sense for me that the AutoML user guide currently focuses on the functionality within AutoMLSearch (we currently list out error callbacks, passing in custom pipelines, etc.). This seems more like a flow of how the user should use EvalML.

I guess TLDR is: for what our docs currently look like, I like this change. Maybe this also means we should separate out our docs and not shove everything into the AutoML guide; we could have a intro user guide that steps through the steps a user would take, and then a dedicated AutoML guide that deep-dives into AutoMLSearch specifics.

Will update this, thank you for the great feedback @freddyaboulton!

bchen1116

LGTM! Agree this should be a breaking change and should discuss what happens with automl.ipynb

Addressed requested changes

freddyaboulton

@angela97lin Looks great! This change add two uncovered lines to our repo.

Can you please add coverage in test_data_checks.py It should be quick but we can also handle that in a separate issue.

In my opinion, only the first uncovered line is important: raise ValueError(f"Parameter data_checks must be a list. Received {type(data_check_classes).__name__}."). We're probably going to get rid of the AutoMLDataChecks class (I can file an issue) since AutoMLSearch no longer knows about data checks so the second uncovered line is not high priority.

freddyaboulton · 2021-03-10T16:26:57Z

docs/source/user_guide/automl.ipynb

+   "source": [
+    "### Data Checks\n",
+    "\n",
+    "Before calling `AutoMLSearch.search`, we should run some sanity checks on our data to ensure that the input data being passed will not run into some common issues before running a potentially time-consuming search. EvalML has various data checks that makes this easy. Each data check will return a collection of warnings and errors if it detects potential issues with the input data. This allows users to inspect their data to avoid confusing errors that may arise during the search process. You can learn about each of the data checks available through our [data checks guide](data_checks.ipynb) \n",


angela97lin · 2021-03-11T20:43:06Z

@freddyaboulton Thanks for catching this! Sucks that codecov let me slip this by but I added a test and just decided to remove AutoMLDataChecks in this PR since it's not really useful anymore / was only used in automl.

docs/source/release_notes.rst

dsherry · 2021-03-12T15:45:16Z

docs/source/user_guide/automl.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Since there were no warnings or errors returned, we can safely continue with the search process."


This is great! Gonna be really cool when we can replace this with the data check actions :D

Agreeeeed 🥳

evalml/data_checks/data_checks.py

evalml/automl/automl_search.py

dsherry

Excellent!

init

5c63570

angela97lin self-assigned this Mar 5, 2021

angela97lin added 2 commits March 5, 2021 14:23

eek

88b9ffc

retrigger for rtd

7e5fb2e

angela97lin changed the title ~~Move data check code from AutoML search to init~~ Delete data check code from AutoML search Mar 8, 2021

hackity hack hack away at automl data check code

6310ef7

angela97lin added 2 commits March 9, 2021 00:33

delete from notebooks and lint

7a2e6c8

Merge branch 'main' into 1882_init

7dd2cc5

angela97lin commented Mar 9, 2021

View reviewed changes

angela97lin requested review from dsherry, freddyaboulton, ParthivNaresh, bchen1116, chukarsten and jeremyliweishih March 9, 2021 16:07

angela97lin marked this pull request as ready for review March 9, 2021 16:07

angela97lin added this to the Sprint 2021 Mar A milestone Mar 9, 2021

freddyaboulton previously requested changes Mar 9, 2021

View reviewed changes

bchen1116 approved these changes Mar 9, 2021

View reviewed changes

Merge branch 'main' into 1882_init

69642ad

dsherry removed this from the Sprint 2021 Mar A milestone Mar 9, 2021

cleanup and address feedback

44df009

angela97lin requested a review from freddyaboulton March 9, 2021 22:44

freddyaboulton approved these changes Mar 10, 2021

View reviewed changes

chukarsten approved these changes Mar 11, 2021

View reviewed changes

remove automldatacheck

dd081d8

add test

dc4913d

angela97lin added 3 commits March 11, 2021 15:44

Merge branch 'main' into 1882_init

569ec9c

release notes

85b6f39

Merge branch 'main' into 1882_init

7ecd5bd

dsherry reviewed Mar 12, 2021

View reviewed changes

docs/source/release_notes.rst Outdated Show resolved Hide resolved

dsherry reviewed Mar 12, 2021

View reviewed changes

evalml/data_checks/data_checks.py Show resolved Hide resolved

dsherry reviewed Mar 12, 2021

View reviewed changes

evalml/automl/automl_search.py Show resolved Hide resolved

dsherry approved these changes Mar 12, 2021

View reviewed changes

angela97lin added 2 commits March 13, 2021 21:25

clean up release notes

f2bb4d7

Merge branch 'main' into 1882_init

b1b04c1

angela97lin merged commit 1b3e084 into main Mar 14, 2021
12 checks passed

angela97lin deleted the 1882_init branch March 14, 2021 04:08

dsherry mentioned this pull request Mar 24, 2021

Release v0.21.0 #2029

Merged

Delete data check code from AutoML search #1935

Delete data check code from AutoML search #1935

angela97lin commented Mar 5, 2021

codecov bot commented Mar 8, 2021 •

edited

angela97lin Mar 9, 2021

freddyaboulton left a comment

freddyaboulton Mar 9, 2021

angela97lin Mar 9, 2021 •

edited

bchen1116 left a comment

freddyaboulton left a comment •

edited

freddyaboulton Mar 10, 2021

angela97lin Mar 11, 2021

angela97lin commented Mar 11, 2021

dsherry Mar 12, 2021

angela97lin Mar 14, 2021

dsherry left a comment

Delete data check code from AutoML search #1935

Delete data check code from AutoML search #1935

Conversation

angela97lin commented Mar 5, 2021

codecov bot commented Mar 8, 2021 • edited

Codecov Report

angela97lin Mar 9, 2021

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

freddyaboulton Mar 9, 2021

Choose a reason for hiding this comment

angela97lin Mar 9, 2021 • edited

Choose a reason for hiding this comment

bchen1116 left a comment

Choose a reason for hiding this comment

freddyaboulton left a comment • edited

Choose a reason for hiding this comment

freddyaboulton Mar 10, 2021

Choose a reason for hiding this comment

angela97lin Mar 11, 2021

Choose a reason for hiding this comment

angela97lin commented Mar 11, 2021

dsherry Mar 12, 2021

Choose a reason for hiding this comment

angela97lin Mar 14, 2021

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 8, 2021 •

edited

angela97lin Mar 9, 2021 •

edited

freddyaboulton left a comment •

edited