Update automl algorithm #793

dsherry · 2020-05-21T16:11:55Z

Fixes #272 fixes #754 fixes #755 fixes #756 fixes #757 fixes #301 fixes #309 fixes #353

Adds a new iterative automl algorithm, and an API for defining automl algorithms in general.

Also updates default parameters for components and pipelines.

API docs are included, but updating the rest of the docs is not included in this PR.

codecov · 2020-05-28T01:12:03Z

Codecov Report

Merging #793 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #793      +/-   ##
==========================================
+ Coverage   99.63%   99.65%   +0.01%     
==========================================
  Files         170      175       +5     
  Lines        6647     6940     +293     
==========================================
+ Hits         6623     6916     +293     
  Misses         24       24

Impacted Files	Coverage Δ
evalml/__init__.py	`100.00% <ø> (ø)`
evalml/automl/auto_classification_search.py	`100.00% <ø> (ø)`
evalml/automl/auto_regression_search.py	`100.00% <ø> (ø)`
...eature_selection/rf_classifier_feature_selector.py	`100.00% <ø> (ø)`
...feature_selection/rf_regressor_feature_selector.py	`100.00% <ø> (ø)`
evalml/tests/tuner_tests/test_grid_search_tuner.py	`100.00% <ø> (ø)`
...alml/tests/tuner_tests/test_random_search_tuner.py	`100.00% <ø> (ø)`
evalml/automl/__init__.py	`100.00% <100.00%> (ø)`
evalml/automl/auto_search_base.py	`97.91% <100.00%> (-0.09%)`	⬇️
evalml/automl/automl_algorithm/__init__.py	`100.00% <100.00%> (ø)`
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26fbdc6...c48df2b. Read the comment docs.

dsherry · 2020-05-28T14:58:31Z

@kmax12 @jeremyliweishih @angela97lin this PR is currently missing some more docstrings, a few more unit tests, and general documentation updates. I will get that stuff in shortly. But otherwise this is ready for review.

kmax12

here's my first pass through. probably best to discuss live anything that need clarity. need to look at tests more in next round.

kmax12 · 2020-05-28T14:37:53Z

evalml/tests/automl_tests/test_iterative_algorithm.py

+    MockEstimator = dummy_classifier_estimator_class
+
+    class MockBinaryClassificationPipeline1(BinaryClassificationPipeline):
+        estimator = MockEstimator


do you need this estimator property?

Yeah. Updated in latest push, and now its clear why.

still not clear to me, but i might not be looking in the right place. the reason why i feel it's not need is because we have this line in PipelineBase.__init__()

self.estimator = self.component_graph[-1] if isinstance(self.component_graph[-1], Estimator) else None

The mock estimator defines the dummy hyperparameter ranges, which although not referred to directly are used in the last unit test to make sure the tuner searched for params which differ from the default.

evalml/automl/automl_algorithm/iterative_algorithm.py

evalml/automl/automl_algorithm/__init__.py

evalml/automl/automl_algorithm/automl_algorithm.py

evalml/automl/automl_algorithm/iterative_algorithm.py

evalml/tests/automl_tests/test_auto_classification_search.py

dsherry · 2020-05-28T17:54:38Z

Just spoke with @kmax12 . Plan is that I'll address most of his comments, then extend the IterativeAlgorithm to be able to continue indefinitely and to respect max_pipelines, because we feel that's the most useful. There's a risk that will take too long for us to get it into the release tomorrow; if that happens we can always cut another next week.

kmax12

proactively leaving some more review comments. looking very close in my eyes!

evalml/automl/automl_algorithm/automl_algorithm.py

kmax12 · 2020-05-29T11:54:54Z

docs/source/automl/pipeline_search.ipynb

-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "automl.possible_pipelines"


should we add this line back in but be automl.allowed_pipelines? it actually might make more sense to just replaces this with print(automl) in the docs to give complete overview

we can come back this in a future PR if we want since it's doc changes

Yeah, since this PR crossed the 1000+ line threshold, I'd like to do docs changes separately. That may mean we cut the release without great documentation for AutoMLAlgorithm/IterativeAlgorithm. This doesn't bother me too much because this PR doesn't change much for users in terms of the inputs to automl. You're reminding me though, I'll scan the docs real quick and make sure the external params like allowed_pipelines and allowed_model_families are covered well.

Eh, I'll push this back in for now, using allowed_pipelines

kmax12 · 2020-05-29T11:58:05Z

evalml/tests/automl_tests/test_iterative_algorithm.py

+    MockEstimator = dummy_classifier_estimator_class
+
+    class MockBinaryClassificationPipeline1(BinaryClassificationPipeline):
+        estimator = MockEstimator


still not clear to me, but i might not be looking in the right place. the reason why i feel it's not need is because we have this line in PipelineBase.__init__()

self.estimator = self.component_graph[-1] if isinstance(self.component_graph[-1], Estimator) else None

evalml/automl/automl_algorithm/iterative_algorithm.py

dsherry · 2020-05-29T13:43:19Z

Thanks @kmax12 ! I was about to message and say that I worked through all your previous comments and this is ready for re-review.

dsherry · 2020-05-29T13:57:05Z

evalml/pipelines/components/estimators/classifiers/rf_classifier.py

@@ -11,12 +11,12 @@ class RandomForestClassifier(Estimator):
    name = "Random Forest Classifier"
    hyperparameter_ranges = {
        "n_estimators": Integer(10, 1000),
-        "max_depth": Integer(1, 32),


@kmax12 I ended up having to update some of the param defaults and ranges, according to what I could find recommended on the sklearn/catboost/xgboost sites. We've had this recurring problem where if calls to the random number generator were changed in a PR, the runtime on a few tests could suddenly spike. I think lowering these max_depth max values for tree models has helped with that. I think its sped up the unit tests overall, which is nice. And we're close to being able to test all of this directly via the perf tests! (@jeremyliweishih )

dsherry · 2020-05-29T14:01:56Z

evalml/pipelines/components/estimators/classifiers/rf_classifier.py

    }
    model_family = ModelFamily.RANDOM_FOREST
    supported_problem_types = [ProblemTypes.BINARY, ProblemTypes.MULTICLASS]

-    def __init__(self, n_estimators=10, max_depth=None, n_jobs=-1, random_state=0):
+    def __init__(self, n_estimators=100, max_depth=6, n_jobs=-1, random_state=0):


An unfortunate limitation of the current approach is that all default values must be in the hyperparam range. This means max_depth here has to default to an integer value. Sklearn's doc says for RF, max_depth's default is None so that its set automatically using some heuristic they have. We should consider how to take advantage of that in the future.

This wasn't a problem before this PR because we never tried to use the default parameters in automl! We always sampled from the hyperparameter ranges.

One option is to acknowledge that the defaults won't necessarily appear in the hyperparam ranges, and update the code to not try to add the evaluation results to the tuner, which is where the error occurs. Another option is to change the max_depth range to be something like [None] + list(range(1, 11)), but then I think the CASH algos won't know anymore that the numeric values are ordered. We can come back to this when its time for more rigorous perf testing (@jeremyliweishih FYI)

dsherry · 2020-05-29T14:03:10Z

evalml/pipelines/components/transformers/feature_selection/rf_classifier_feature_selector.py

@@ -20,7 +20,8 @@ def __init__(self, number_features=None, n_estimators=10, max_depth=None,
        if number_features:
            max_features = max(1, int(percent_features * number_features))
        parameters = {"percent_features": percent_features,
-                      "threshold": threshold}
+                      "threshold": threshold,
+                      "number_features": number_features}


#819 removed the feature selectors from all pipelines. But in order to get the unit tests from that PR passing, and preserve support for feature selectors in automl, the feature selectors must save the number_features parameter.

dsherry · 2020-05-29T14:05:46Z

evalml/tests/automl_tests/test_auto_classification_search.py

-    automl_1 = AutoClassificationSearch(objective=fc, max_pipelines=5, random_state=30)
-    automl_1.search(X, y)
-
-    assert automl.rankings.equals(automl_1.rankings)


I removed this because this unit test was taking way too long, something like 90 seconds! This test just needs to run two automl searches and check the results are identical. We have separate coverage that FraudCost is a valid objective. I think this is left over from before the objective API overhaul.

dsherry · 2020-05-29T14:06:35Z

evalml/tests/automl_tests/test_auto_classification_search.py

+@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
+def test_automl_allowed_pipelines_search(mock_fit, mock_score, dummy_binary_pipeline_class, X_y):
+    X, y = X_y
+    mock_score.return_value = {'Log Loss Binary': 1.0}


I'm getting pretty into this mocking pattern to save time in the automl tests. Any time the results of pipelines' fit and score methods don't matter, its great to mock them out to save time in the search.

dsherry · 2020-05-29T14:24:16Z

evalml/tests/automl_tests/test_autobase.py

@@ -315,11 +323,64 @@ def fit(self, X, y):
    mock_get_pipelines.return_value = allowed_pipelines
    start_iteration_callback = MagicMock()
    automl = AutoClassificationSearch(max_pipelines=2, start_iteration_callback=start_iteration_callback)
-    assert automl.possible_pipelines == allowed_pipelines


Now covered in test_automl_allowed_pipelines_init in classification/regression tests

dsherry · 2020-05-29T14:37:10Z

evalml/tuners/skopt_tuner.py

+                    .format(pipeline_parameters, str(e))
+                logger.error(msg)
+                raise ParameterError(msg)
+            raise(e)


I added this code to handle the case when someone passes in a value like None but the range is Int(1, 32). In that case an ugly stack trace gets thrown. This makes the error more pretty. I also avoided the error in automl search by changing our default params to not use None for max_depth

…families and pipelines from AutoSearchBase

…families

…d tests, updated doc

dsherry · 2020-05-29T19:01:43Z

docs/source/api_reference.rst

+    :nosignatures:
+
+    AutoMLAlgorithm
+    IterativeAlgorithm


@angela97lin how does this look for now?

dsherry · 2020-05-29T19:06:06Z

evalml/automl/auto_search_base.py

+                    current_batch_pipelines = self._automl_algorithm.next_batch()
+                except StopIteration:
+                    logger.info('AutoML Algorithm out of recommendations, ending')
+                    break


I will update this code in a subsequent PR. Left over from when I had the algo throwing StopIteration. This doesn't block from merging, and has test coverage in place, just that we should probably delete it to simplify and instead catch AutoMLAlgorithmError

dsherry mentioned this pull request May 21, 2020

Update AutoBase with data_checks parameter #765

Merged

dsherry force-pushed the ds_754_automl_algorithm branch from 6649556 to f6f1f73 Compare May 21, 2020 23:33

dsherry mentioned this pull request May 22, 2020

Automl data_checks parameter: how best to support disable vs default vs custom #799

Closed

dsherry force-pushed the ds_754_automl_algorithm branch 3 times, most recently from d26f893 to 8889796 Compare May 27, 2020 23:38

dsherry marked this pull request as ready for review May 28, 2020 14:56

dsherry requested review from angela97lin, jeremyliweishih and kmax12 May 28, 2020 14:56

auto-assign bot assigned dsherry May 28, 2020

kmax12 suggested changes May 28, 2020

View reviewed changes

dsherry force-pushed the ds_754_automl_algorithm branch from 0535acd to 466d816 Compare May 28, 2020 16:27

dsherry force-pushed the ds_754_automl_algorithm branch from 0b81bd3 to 0a9a863 Compare May 28, 2020 23:50

kmax12 suggested changes May 29, 2020

View reviewed changes

dsherry commented May 29, 2020

View reviewed changes

dsherry force-pushed the ds_754_automl_algorithm branch 2 times, most recently from d3eeead to 2f93b89 Compare May 29, 2020 15:10

dsherry requested a review from kmax12 May 29, 2020 15:10

dsherry mentioned this pull request May 29, 2020

Add ExtraTrees Pipeline #790

Merged

dsherry added 20 commits May 29, 2020 13:47

Lint

cbf635f

Fixed two broken tests

92cc18a

Update automl algo to not use objective, and to expect allowed model …

03fc589

…families and pipelines from AutoSearchBase

Finish updating automl classes to accept allowed pipelines and model …

dd2d5ed

…families

Move copyright to base __init__.py

2a1b5c8

Remove allowed_model_families from automl algorithm classes, not needed

23aabb3

Missed a spot in prev commit

8e3f75f

Simplify logic for allowed_pipelines and allowed_model_families. Adde…

c40e0e9

…d tests, updated doc

Remove unused method

f23866f

Call handle_model_family on model families, to handle str inputs

97ff97a

Have the automl algo continue indefinitely

b883ce1

Add back in some old coverage of automl str method

5164d93

Remove unused logger change

c55ae2c

Add test for codecov

8036c90

Add test coverage for codecov

40f1f2d

Save automl algo instance as instance state for debugging

c03c955

Docs fix

9100229

Update allowed_pipelines docs

7f4919d

Add a simple class-level docstring for autosearchbase

aab30fc

Add automl algo classes to API ref

de9c9ae

dsherry mentioned this pull request May 29, 2020

Add Elastic Net Pipeline #812

Merged

dsherry force-pushed the ds_754_automl_algorithm branch from 2f93b89 to de9c9ae Compare May 29, 2020 18:33

dsherry added 3 commits May 29, 2020 14:41

Add AutoSearchBase to init so it can be included in docs

c5be618

Add back in showing the allowed pipelines in docs

d6637e4

Update changelog

c48df2b

dsherry commented May 29, 2020

View reviewed changes

dsherry merged commit bbc802a into master May 29, 2020

dsherry deleted the ds_754_automl_algorithm branch May 29, 2020 19:22

dsherry mentioned this pull request May 29, 2020

Update docs for automl algorithm #823

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update automl algorithm #793

Update automl algorithm #793

dsherry commented May 21, 2020 •

edited

Loading

codecov bot commented May 28, 2020 •

edited

Loading

dsherry commented May 28, 2020

kmax12 left a comment

kmax12 May 28, 2020

dsherry May 28, 2020

kmax12 May 29, 2020

dsherry May 29, 2020

dsherry commented May 28, 2020

kmax12 left a comment

kmax12 May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

kmax12 May 29, 2020

dsherry commented May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020

dsherry May 29, 2020 •

edited

Loading

angela97lin May 29, 2020

dsherry May 29, 2020

Update automl algorithm #793

Update automl algorithm #793

Conversation

dsherry commented May 21, 2020 • edited Loading

codecov bot commented May 28, 2020 • edited Loading

Codecov Report

dsherry commented May 28, 2020

kmax12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry commented May 28, 2020

kmax12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry commented May 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry May 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry commented May 21, 2020 •

edited

Loading

codecov bot commented May 28, 2020 •

edited

Loading

dsherry May 29, 2020 •

edited

Loading