AutoMLSearch: hold reference to training data by dsherry · Pull Request #1597 · alteryx/evalml

dsherry · 2020-12-23T01:32:49Z

Changes

Update AutoMLSearch constructor to take training data as X_train and y_train. Update search and add_to_leaderboard to no longer require data
Remove the woodwork warnings when pandas dtypes are provided (@angela97lin heads up this will cause mild conflicts with Allow users to set feature types without having to learn about woodwork directly #1555 . I can try to back it out of this PR if you'd like, but it felt easier in the moment to just remove here heh)
Update split_data helper to accept problem_type, problem_configuration, test_size, shuffle, and to default random_state to 0
Rename the "Testing" split (during CV) in the automl log output to "Validation"
Move evalml/automl/data_splitters/ to evalml/preprocessing/data_splitters/ to avoid circular import
Add problem type utils: is_regression, is_classification, is_timeseries, is_binary, is_multiclass
Update documentation

codecov · 2020-12-23T03:49:43Z

Codecov Report

Merging #1597 (05a6c74) into main (54bf169) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1597     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         240      240             
  Lines       18056    18092     +36     
=========================================
+ Hits        18048    18084     +36     
  Misses          8        8

Impacted Files	Coverage Δ
evalml/automl/__init__.py	`100.0% <ø> (ø)`
evalml/preprocessing/data_splitters/__init__.py	`100.0% <ø> (ø)`
.../preprocessing/data_splitters/time_series_split.py	`100.0% <ø> (ø)`
evalml/automl/automl_search.py	`99.7% <100.0%> (-<0.1%)`	⬇️
evalml/automl/utils.py	`100.0% <100.0%> (ø)`
evalml/preprocessing/__init__.py	`100.0% <100.0%> (ø)`
...essing/data_splitters/training_validation_split.py	`100.0% <100.0%> (ø)`
evalml/preprocessing/utils.py	`100.0% <100.0%> (ø)`
evalml/problem_types/__init__.py	`100.0% <100.0%> (ø)`
evalml/problem_types/utils.py	`100.0% <100.0%> (ø)`
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54bf169...05a6c74. Read the comment docs.

dsherry · 2020-12-23T22:09:41Z


-        if not isinstance(y, ww.DataColumn):
-            logger.warning("`y` passed was not a DataColumn. EvalML will try to convert the input as a Woodwork DataTable and types will be inferred. To control this behavior, please pass in a Woodwork DataTable instead.")
-            y = _convert_to_woodwork_structure(y)


I removed these warnings because we were planning on doing so (#1555) and because it was easier than fixing the failing test coverage for them 😅

dsherry · 2020-12-23T22:10:14Z

        return {objective_name: float(score) / n_folds for objective_name, score in scores.items()}

-    def _compute_cv_scores(self, pipeline, X, y):
+    def _compute_cv_scores(self, pipeline):


It would be great to get more eyes on the changes here. I think I did it right but can't hurt to check.

dsherry · 2020-12-23T22:10:39Z

            ordered_scores.update(scores)
            ordered_scores.update({"# Training": y_train.shape[0]})
-            ordered_scores.update({"# Testing": y_test.shape[0]})
+            ordered_scores.update({"# Validation": y_valid.shape[0]})


I renamed this field because I think "validation" is a better description

dsherry · 2020-12-23T22:11:21Z

            gap (int): Gap used in time series problem. Time series pipelines shift the target variable by gap rows
                since we are interested in
+            n_splits (int): number of data splits to make.
+            test_size (float): What percentage of data points should be included in the test set. Defaults to None.


Oops, I will remove test_size, left over

dsherry · 2020-12-23T22:12:35Z

        """
        train, test = train_test_split(np.arange(X.shape[0]), test_size=self.test_size, train_size=self.train_size, shuffle=self.shuffle, stratify=self.stratify, random_state=self.random_state)
-        return [(train, test)]
+        return iter([(train, test)])


I believe this was necessary because BaseCrossValidator subclasses are expected to return iterators. Some tests failed without it. I had to update other tests to call list(...) 🤷‍♂️

dsherry · 2020-12-23T22:13:27Z

-    train, test = next(CV_method.split(X.to_dataframe(), y.to_series()))
+    data_splitter = None
+    if is_time_series(problem_type):
+        data_splitter = TrainingValidationSplit(test_size=test_size, shuffle=False, stratify=None, random_state=random_state)


This is the change @freddyaboulton and I were discussion. Once this PR is merged we can update looking glass to use split_data

bchen1116

LGTM! Left a couple comments but I think you caught everything!

kmax12 · 2020-12-28T22:19:10Z

+    "\n",
+    "Below EvalML utilizes Bayesian optimization (EvalML's default optimizer) to search and find the best pipeline defined by the given objective.\n",
+    "\n",
+    "EvalML provides a number of parameters to control the search process. `max_batches` is one of the parameters which controls the stopping criterion for the AutoML search. It indicates the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. In this example, `max_batches` is set to 1.\n",


it might be better to move the explanation of max_batches until after the code example. it doesn't mean anything to see that parameter defined until it's actually used. for example, see in the featuretools docs how we 1. show the function 2. explain what the function did 3. explain the parameters that controlled the function.

not sure it's helpful to introduce the word "round" here since it's basically functioning as a synonym for batch without actually defining the word batch.

We also don't give the user any intuition of when or why they'd want to use the parameter i.e it controls the "budget" evalml has to look for the best model. increasing the value allows evalml to search deeper for the best model, at the risk of overfitting.

i see that this text wasn't actually change in the PR, so maybe best done as a follow up

…size, shuffle, random_state

…turn iterator

…upervised learning

* First round * Removed RMSLE, MSLE, and MAPE from non_core_objectives * Add objective parameter to all data_check subclasses * Add new data_check_message_code for a target that is incompatible with an objective * Add check to invalid_target_data_check for RMSLE, MSLE, and MAPE * Invalid Target Data Check update * Fix test case to not include invalid objectives in regression_core_objectives * Test updates * Invalid target data check update * Lint * Release notes * Tests and updates to code * Release notes * Latest Dependencies Version update * Lint fix * Dependency check * Fixing minor issues * test none objective update * Update to reflect changes in #1597 * Make objective mandatory for DefaultDataChecks and InvalidTargetDataCheck * Mock data check to see if objective is being passed from AutoML * Fix breaking tests due to mandatory objective param in DefaultDataChecks, and check error output from DefaultDataChecks * Change label to all positive so RMSLE, MSLE, and MAPE can be included in regression_core_objectives * Raise error for None objective in InvalidTargetDataCheck * Docstring example fix * Lint and docstring example * Jupyter notebook update for build_docs * Jupyter notebook update * remove print statement * Added test cases and docstring, updated error message * test case fixes * Test update * lint error * Add positive_only class attribute for ObjectiveBase * Adding @classproperty to ObjectiveBase for positive_only check * Lint error * Fix conflicts with #1551 * Lint

dsherry added the enhancement An improvement to an existing feature. label Dec 23, 2020

dsherry marked this pull request as ready for review December 23, 2020 21:30

auto-assign Bot assigned dsherry Dec 23, 2020

dsherry commented Dec 23, 2020

View reviewed changes

dsherry requested review from ParthivNaresh, angela97lin, bchen1116, freddyaboulton and jeremyliweishih December 23, 2020 22:13

dsherry added this to the December 2020 milestone Dec 23, 2020

bchen1116 approved these changes Dec 28, 2020

View reviewed changes

Comment thread docs/source/start.ipynb Outdated

Comment thread evalml/problem_types/utils.py

kmax12 reviewed Dec 28, 2020

View reviewed changes

Comment thread README.md Outdated

kmax12 reviewed Dec 28, 2020

View reviewed changes

dsherry added 11 commits December 29, 2020 09:27

Add is_regression, is_classification, is_timeseries prob type helpers

19a7345

Update TS splitter to take test_size arg

295a424

Update split_data method to accept problem type, configuration, test_…

a0244b5

…size, shuffle, random_state

Update AutoMLSearch to accept data in the constructor

7fcfd16

Lint

885250d

Update automl tests

522d705

Move data splitters into preprocessing

1f48906

Update tests

ddf9232

Update docs

23217f3

Fix lint errors in tests

fbea1dd

Fix formatting errors in docs

85ddace

dsherry added 21 commits December 29, 2020 09:27

Fix another test

c678cc2

Fix another test

ce9259d

Remove shuffle arg from split_data, don't need to support that right now

8865924

Docs updates for split_data

90d7ae7

Missed a spot with split_data shuffle revert

c7e2136

Test cleanup

ffedf16

Fix tests

2cc1453

Fix a test

2f1b585

Missed a spot with split_data, fixing

eef34ef

Fix another test

bfc4f3c

Fix another test

e27a5c6

Fix a couple more tests

853ae5d

Release notes

a04ec73

Move tv split tests

593c79f

Add is_binary and is_multiclass problem type helpers

78b38e7

Use TrainingValidationSplit in split_data. Update that splitter to re…

21d00b9

…turn iterator

Lint

b92ca04

Remove the addition of holdout from this PR

6f1cd23

Remove unneeded docs change

cdbea04

Make X/y keyword args instead of named, for future flexibility w/ uns…

d95bc70

…upervised learning

Unit tests for prob type utils

49c3038

dsherry force-pushed the ds_blitz_automl branch from 3bdad96 to 49c3038 Compare December 29, 2020 15:32

Always use keyword arg names for X_train/y_train

05a6c74

dsherry merged commit edb6ed6 into main Dec 29, 2020

dsherry deleted the ds_blitz_automl branch December 29, 2020 16:20

This was referenced Dec 29, 2020

Compute threshold tuning split in AutoMLSearch constructor #1621

Open

Return trained pipeline for AutoMLSearch.best_pipeline #1547

Merged

ParthivNaresh added a commit that referenced this pull request Dec 29, 2020

Update to reflect changes in #1597

9c368d6

dsherry mentioned this pull request Dec 29, 2020

Release v0.17.0 #1623

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AutoMLSearch: hold reference to training data#1597

AutoMLSearch: hold reference to training data#1597
dsherry merged 35 commits into
mainfrom
ds_blitz_automl

dsherry commented Dec 23, 2020 •

edited

Loading

Uh oh!

codecov Bot commented Dec 23, 2020 •

edited

Loading

Uh oh!

dsherry Dec 23, 2020

Uh oh!

dsherry Dec 23, 2020

Uh oh!

dsherry Dec 23, 2020

Uh oh!

dsherry Dec 23, 2020

Uh oh!

dsherry Dec 23, 2020 •

edited

Loading

Uh oh!

dsherry Dec 23, 2020

Uh oh!

bchen1116 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmax12 Dec 28, 2020 •

edited

Loading

Uh oh!

kmax12 Dec 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dsherry commented Dec 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Dec 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dsherry Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

dsherry Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

dsherry Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

dsherry Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

dsherry Dec 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dsherry Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kmax12 Dec 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kmax12 Dec 28, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dsherry commented Dec 23, 2020 •

edited

Loading

codecov Bot commented Dec 23, 2020 •

edited

Loading

dsherry Dec 23, 2020 •

edited

Loading

kmax12 Dec 28, 2020 •

edited

Loading