Better default value for data_checks in AutoSearchBase #892

freddyaboulton · 2020-06-26T19:10:19Z

Pull Request Description

Previously, it was awkward to disable data checking in AutoSearchBase.search because the default value was None and it corresponded to DefaultDataChecks. This PR addresses #799 by letting the data_checks parameter be a list, str, DataChecks, or None to give the user more control over the data checks they want to perform.

After creating the pull request: in order to pass the changelog_updated check you will need to update the "Future Release" section of docs/source/changelog.rst to include this pull request by adding :pr:123.

… github.com:FeatureLabs/evalml into 799-better-handling-of-default-vs-disable-vs-custom

codecov · 2020-06-26T19:21:10Z

Codecov Report

Merging #892 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #892   +/-   ##
=======================================
  Coverage   99.77%   99.77%           
=======================================
  Files         193      193           
  Lines        8851     8881   +30     
=======================================
+ Hits         8831     8861   +30     
  Misses         20       20

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`98.45% <100.00%> (+0.04%)`	⬆️
evalml/data_checks/data_checks.py	`100.00% <100.00%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d83c70f...7dcb9ce. Read the comment docs.

freddyaboulton · 2020-06-26T19:36:12Z

@dsherry @angela97lin Fixed the diff, not sure what happened. Sorry about that!

dsherry · 2020-06-26T20:01:19Z

@freddyaboulton lol no problem! Thanks, will take a look.

dsherry

@freddyaboulton functionality and tests look good! Left a comment about splitting the validation between DataChecks and automl search. Once we resolve that conversation I'll approve.

dsherry · 2020-06-26T20:00:55Z

docs/source/changelog.rst

@@ -20,12 +20,13 @@ Changelog
        * Added SelectColumns transformer :pr:`873`
        * Added ability to evaluate additional pipelines for automl search :pr:`874`
        * Added `default_parameters` class property to components and pipelines :pr:`879`
+        * Better support for disabling data checks in automl search :pr:`892`


We should add something to the breaking changes section in here as well, because we're deleting EmptyDataChecks and changing the pattern

We should also rephrase this as "Added better support..." w.r.t our contribution docs!

dsherry · 2020-06-26T20:03:01Z

evalml/automl/automl_search.py

-        if not isinstance(data_checks, DataChecks):
-            raise ValueError("data_checks parameter must be a DataChecks object!")
-
+        data_checks = self._validate_data_checks(data_checks)


Let's have the DataCheck object do the list validation part for us, but do the string parsing and stuff here, because that's specific to automl search. To steal from what you've already got:

if data_checks == 'auto': data_checks = DefaultDataChecks() elif data_checks == 'disabled': data_checks = EmptyDataChecks() # could still use this internally elif isinstance(data_checks, list): data_checks = DataChecks(data_checks) elif not isinstance(data_checks, DataChecks): raise AutoMLSearchException('invalid value of type {} provided for "data_checks" parameter'.format(...))

And then we update DataChecks.__init__ to validate the list input, as you have above.

The nice thing about this is that our DataChecks can take a list right now, so we'd be able to reuse the list validation code in the future.

Thoughts?

Sounds good to me! Makes sense to move the list validaton to DataChecks since its already expecting a list.

dsherry · 2020-06-26T20:14:30Z

evalml/tests/automl_tests/test_automl.py

+        automl.search(X, y, data_checks=[1])
+    with pytest.raises(ValueError, match="If data_checks is a string, it must be either 'auto' or 'disabled'. "
+                                         "Received 'default'."):
+        automl.search(X, y, data_checks="default")


dsherry · 2020-06-26T20:14:55Z

evalml/tests/automl_tests/test_automl.py


+    _run_test(with_data_checks=[MockDataCheckErrorAndWarning()])
+    _run_test(with_data_checks=DataChecks([MockDataCheckErrorAndWarning()]))


IMO its fine to scope this public since its test code

Also, perhaps instead of this you could use @pytest.mark.parametrize, since that's a pattern we've started using? I think its a bit easier to read but I acknowledge that's an opinion on style and not a functional difference.

I'll use parametrize!

angela97lin · 2020-06-26T20:55:38Z

evalml/automl/automl_search.py

+                raise an exception.
+
+        Returns:
+            An instance of DataChecks to perform.


"An instance of DataChecks used to perform checks before search..." or something along those lines?

angela97lin · 2020-06-26T20:58:01Z

evalml/tests/automl_tests/test_automl.py

+    automl.search(X, y, data_checks="disabled")
+    assert automl.data_check_results is None


Duplicate of above?

angela97lin · 2020-06-26T20:59:40Z

evalml/tests/automl_tests/test_automl.py

        automl.search(X, y, data_checks=1)
+    with pytest.raises(ValueError, match="All elements of parameter data_checks must be an instance of DataCheck."):


Maybe silly and unnecessary but could be good to add a test where one element is a data check but one element isn't? Ex: [EmptyDataChecks(), 1].

…heck.

…vs-custom

freddyaboulton · 2020-06-26T22:53:04Z

@dsherry @angela97lin I have addressed your comments!

dsherry

Awesome!

dsherry · 2020-06-29T15:04:48Z

evalml/automl/automl_search.py

+
+        if isinstance(data_checks, DataChecks):
+            return data_checks
+


Silly style nit-pick: delete the newlines to keep the file smaller? 😂

dsherry · 2020-06-29T15:12:46Z

evalml/automl/automl_search.py

@@ -241,10 +275,12 @@ def search(self, X, y, data_checks=None, feature_types=None, raise_errors=True,
            show_iteration_plot (boolean, True): Shows an iteration vs. score plot in Jupyter notebook.
                Disabled by default in non-Jupyter enviroments.

-            data_checks (DataChecks, None): A collection of data checks to run before searching for the best classifier. If data checks produce any errors, an exception will be thrown before the search begins. If None, uses DefaultDataChecks. Defaults to None.
+            data_checks (DataChecks, list(Datacheck), str, None): A collection of data checks to run before searching
+                for the best classifier. If data checks produce any errors, an exception will be thrown before the


Instead of saying "classifier" can say "A collection of data checks to run before automl search."

dsherry · 2020-06-29T15:14:18Z

evalml/data_checks/data_checks.py

@@ -8,6 +11,10 @@ def __init__(self, data_checks=None):
        Arguments:
            data_checks (list (DataCheck)): list of DataCheck objects
        """
+
+        if not all(isinstance(check, DataCheck) for check in data_checks):
+            raise ValueError("All elements of parameter data_checks must be an instance of DataCheck.")


Looks good.

Can't we also move the list check in here?

dsherry · 2020-06-29T15:15:42Z

evalml/tests/automl_tests/test_automl.py

+    assert automl.data_check_results is None
+
+    automl.search(X, y, data_checks=None)
+    assert automl.data_check_results is None


@freddyaboulton would this dataset normally return data check results? If not, we should find a dataset or mocking pattern to ensure that's the case, so that this test proves the checks are in fact disabled here.

Good point! I don't think this dataset would return any check errors. I'll make the necessary changes.

dsherry · 2020-06-29T15:16:30Z

evalml/tests/automl_tests/test_automl.py

+
+@pytest.mark.parametrize("data_checks",
+                         [[MockDataCheckErrorAndWarning()],
+                          DataChecks([MockDataCheckErrorAndWarning()])])


dsherry · 2020-06-29T15:17:25Z

evalml/tests/automl_tests/test_automl.py

+@pytest.mark.parametrize("data_checks",
+                         [[MockDataCheckErrorAndWarning()],
+                          DataChecks([MockDataCheckErrorAndWarning()])])
+def test_automl_data_checks_raises_error(data_checks, caplog):


Just a note that by default we should mock fit/score on automl tests which run search, if its not too much effort to add

Definitely!

freddyaboulton added 6 commits June 26, 2020 13:01

data_checks can now be a list, str, DataChecks, or None.

1c1913e

Updating changelog for PR 890

89ed6ca

data_checks can now be a list, str, DataChecks, or None.

dbf5b5f

Updating changelog for PR 890

7098e30

Merge branch '799-better-handling-of-default-vs-disable-vs-custom' of…

e19464c

… github.com:FeatureLabs/evalml into 799-better-handling-of-default-vs-disable-vs-custom

Updating changelog for PR 892.

f70c00a

freddyaboulton changed the title ~~799 better handling of default vs disable vs custom~~ Better default value for data_checks in AutoSearchBase Jun 26, 2020

freddyaboulton self-assigned this Jun 26, 2020

freddyaboulton linked an issue Jun 26, 2020 that may be closed by this pull request

Automl data_checks parameter: how best to support disable vs default vs custom #799

Closed

freddyaboulton marked this pull request as ready for review June 26, 2020 19:35

freddyaboulton requested review from angela97lin and dsherry June 26, 2020 19:35

dsherry suggested changes Jun 26, 2020

View reviewed changes

angela97lin reviewed Jun 26, 2020

View reviewed changes

freddyaboulton added 4 commits June 26, 2020 17:55

DataChecks now validate input list to ensure every element is a DataC…

24dd47c

…heck.

Merge branch 'master' into 799-better-handling-of-default-vs-disable-…

c978e71

…vs-custom

Fixing lint issues related to data checks in automl search

1f24a3d

Updating changelog for 892.

f67e15e

freddyaboulton requested a review from dsherry June 29, 2020 14:02

dsherry approved these changes Jun 29, 2020

View reviewed changes

DataChecks now checks if the input parameter is a list.

7dcb9ce

freddyaboulton merged commit 96aecd8 into master Jun 29, 2020

freddyaboulton deleted the 799-better-handling-of-default-vs-disable-vs-custom branch June 29, 2020 16:45

freddyaboulton added this to the June 2020 milestone Jun 29, 2020

angela97lin mentioned this pull request Jun 30, 2020

Release v0.11.0 #901

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better default value for data_checks in AutoSearchBase #892

Better default value for data_checks in AutoSearchBase #892

freddyaboulton commented Jun 26, 2020

codecov bot commented Jun 26, 2020 •

edited

Loading

freddyaboulton commented Jun 26, 2020 •

edited

Loading

dsherry commented Jun 26, 2020

dsherry left a comment

dsherry Jun 26, 2020

angela97lin Jun 26, 2020

dsherry Jun 26, 2020

freddyaboulton Jun 26, 2020

dsherry Jun 26, 2020

dsherry Jun 26, 2020

dsherry Jun 26, 2020

freddyaboulton Jun 26, 2020

angela97lin Jun 26, 2020

angela97lin Jun 26, 2020

angela97lin Jun 26, 2020

freddyaboulton Jun 26, 2020

freddyaboulton commented Jun 26, 2020

dsherry left a comment

dsherry Jun 29, 2020

freddyaboulton Jun 29, 2020

dsherry Jun 29, 2020

dsherry Jun 29, 2020

freddyaboulton Jun 29, 2020

dsherry Jun 29, 2020

freddyaboulton Jun 29, 2020

dsherry Jun 29, 2020

dsherry Jun 29, 2020

freddyaboulton Jun 29, 2020


		_run_test(with_data_checks=[MockDataCheckErrorAndWarning()])
		_run_test(with_data_checks=DataChecks([MockDataCheckErrorAndWarning()]))

		automl.search(X, y, data_checks="disabled")
		assert automl.data_check_results is None

		automl.search(X, y, data_checks=1)
		with pytest.raises(ValueError, match="All elements of parameter data_checks must be an instance of DataCheck."):

Better default value for data_checks in AutoSearchBase #892

Better default value for data_checks in AutoSearchBase #892

Conversation

freddyaboulton commented Jun 26, 2020

Pull Request Description

codecov bot commented Jun 26, 2020 • edited Loading

Codecov Report

freddyaboulton commented Jun 26, 2020 • edited Loading

dsherry commented Jun 26, 2020

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton commented Jun 26, 2020

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 26, 2020 •

edited

Loading

freddyaboulton commented Jun 26, 2020 •

edited

Loading