Integrate Time Series Classification Pipelines into AutoMLSearch #1666

freddyaboulton · 2021-01-07T22:36:25Z

Pull Request Description

Update _set_data_split to choose TimeSeriesSplit for time series classification pipelines
Set log loss as the default objective for time series classification problems
Update the baseline time series regressor to work for classification problems
See if we can delete test_score_works_with_estimator_uses_y in test_time_series_pipeline.py and reuse the tests in test_time_series_baseline_regression.py. See https://github.com/alteryx/evalml/pull/1651/files#r553425518

Minor cosmetic changes that make the diff bigger than it is:

Rename TimeSeriesBaselineRegressor to TimeSeriesBaselineEstimator since it works for all time series problem types
Move evalml/pipelines/regression/time_series_baseline_regression.py to evalml/pipelines/time_series_baselines.py since it contains the definitions for all ts baseline pipelines.
Move evalml/tests/pipeline_tests/regression_pipeline_tests/test_time_series_baseline_regression.py to evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

freddyaboulton · 2021-01-07T23:00:10Z

evalml/tests/automl_tests/test_automl.py

    assert problem_config == {"max_delay": 2, "gap": 3}


-@patch('evalml.pipelines.TimeSeriesRegressionPipeline.score', return_value={"R2": 0.3})


Moving this test to test_automl_search_regression.py

codecov · 2021-01-07T23:41:55Z

Codecov Report

Merging #1666 (c7fcabe) into main (ba7590f) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1666     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         240      240             
  Lines       18625    18652     +27     
=========================================
+ Hits        18617    18644     +27     
  Misses          8        8

Impacted Files	Coverage Δ
evalml/pipelines/components/__init__.py	`100.0% <ø> (ø)`
evalml/pipelines/components/estimators/__init__.py	`100.0% <ø> (ø)`
evalml/pipelines/regression/__init__.py	`100.0% <ø> (ø)`
.../tests/pipeline_tests/test_time_series_pipeline.py	`100.0% <ø> (ø)`
evalml/automl/automl_search.py	`99.7% <100.0%> (+0.1%)`	⬆️
evalml/automl/utils.py	`100.0% <100.0%> (ø)`
evalml/pipelines/__init__.py	`100.0% <100.0%> (ø)`
...lines/components/estimators/regressors/__init__.py	`100.0% <100.0%> (ø)`
...ators/regressors/time_series_baseline_estimator.py	`100.0% <100.0%> (ø)`
evalml/pipelines/time_series_baselines.py	`100.0% <100.0%> (ø)`
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba7590f...c7fcabe. Read the comment docs.

freddyaboulton · 2021-01-08T17:59:03Z

evalml/pipelines/utils.py

        pp_components.append(DateTimeFeaturizer)

-    if problem_type in [ProblemTypes.TIME_SERIES_REGRESSION]:
+    if is_time_series(problem_type):


So that AutoML can create pipelines for ts classification when allowed_pipelines=None

freddyaboulton · 2021-01-08T17:59:31Z

evalml/pipelines/time_series_baselines.py

+)
+
+
+class TimeSeriesBaselineRegressionPipeline(TimeSeriesRegressionPipeline):


Consolidating all of the baseline pipelines for ts into the same file

I like this

freddyaboulton · 2021-01-08T18:00:23Z

evalml/tests/pipeline_tests/test_time_series_pipeline.py

-@pytest.mark.parametrize("pipeline_class", [TimeSeriesBinaryClassificationPipeline,
-                                            TimeSeriesMulticlassClassificationPipeline])
-@pytest.mark.parametrize("use_none_X", [True, False])
-def test_score_works_with_estimator_uses_y(use_none_X, pipeline_class, X_y_binary, X_y_multi):


We have coverage for this case in test_time_series_baseline_pipeline.py

chukarsten

Just some small, comment related stuff. Looks good though. Hefty, hefty.

evalml/automl/automl_search.py

evalml/pipelines/components/estimators/regressors/time_series_baseline_estimator.py

chukarsten · 2021-01-08T20:36:58Z

evalml/pipelines/components/estimators/regressors/time_series_baseline_estimator.py


+    def predict_proba(self, X, y=None):
+        if y is None:
+            raise ValueError("Cannot predict Time Series Baseline Regressor if y is None")


Regressor -> Estimator?

chukarsten · 2021-01-08T20:56:06Z

evalml/tests/pipeline_tests/test_pipelines.py

 )
-from evalml.preprocessing.utils import is_time_series
-from evalml.problem_types import ProblemTypes
+from evalml.problem_types import ProblemTypes, is_time_series


That's probably a smart move.

ParthivNaresh

Looks good to me! Just a few docstring updates

ParthivNaresh · 2021-01-10T22:29:28Z

evalml/pipelines/time_series_baselines.py

+)
+
+
+class TimeSeriesBaselineRegressionPipeline(TimeSeriesRegressionPipeline):


I like this

ParthivNaresh · 2021-01-10T23:10:48Z

evalml/pipelines/components/estimators/regressors/time_series_baseline_estimator.py

+
    @property
    def feature_importance(self):
        """Returns importance associated with each feature. Since baseline regressors do not use input features to calculate predictions, returns an array of zeroes.


Might need to change regressors to estimators here as well

ParthivNaresh · 2021-01-10T23:42:43Z

evalml/pipelines/time_series_classification_pipelines.py

+
+        # In case gap is 0 and this is a baseline pipeline, we drop the nans in the
+        # predictions before decoding them
+        predictions = pd.Series(self._decode_targets(predictions.dropna()), name=self.input_target_name)


Good catch!

bchen1116

Not sure if this is supposed to work currently, but

Additionally, something that we didn't catch previously, but we should change the datachecks we use when we are doing time series problems, specifically the class_imbalance_data_check since TimeSeriesSplit doesn't necessarily shuffle/stratify the data that the other data splitters do:

This can definitely be filed as a separate issue.

The rest looks good to me!

bchen1116 · 2021-01-12T15:38:26Z

evalml/pipelines/components/estimators/regressors/time_series_baseline_estimator.py

+        y = _convert_woodwork_types_wrapper(y.to_series())
+        preds = self.predict(X, y).dropna(axis=0, how='any').astype('int')
+        proba_arr = np.zeros((len(preds), y.max() + 1))
+        proba_arr[np.arange(len(preds)), preds] = 1


bchen1116 · 2021-01-12T16:09:48Z

evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

+                                     "Time Series Baseline Estimator": {'gap': gap, 'max_delay': 1}})
+    expected_y = y
+    if gap == 0:
+        expected_y = y.shift(1)


could shorten this to

expected_y = y.shift(1) if gap == 0 else y

freddyaboulton · 2021-01-12T19:36:30Z

@bchen1116 Thanks for the comments! I filed #1681 to track updates to the class imbalance datacheck. The issue with the fraud dataset is closely related to #1507 because the problem is that the pipelines are delaying categorical features (which introduces NaNs) and then trying to OHE the delayed categorical columns (which can't handle NaNs). I think the problem is that we need a way of identifying which features depend on time and which don't: For fraud we may want to have a record of the currencies used in the last k transactions, but we shouldn't delay features which don't depend on time (like the customer_id if present). I'll add some notes to #1507, we may have to file a separate issue.

…n api_ref.

…timator.py

…ine_pipeline.py

freddyaboulton commented Jan 7, 2021

View reviewed changes

freddyaboulton self-assigned this Jan 8, 2021

freddyaboulton added the enhancement An improvement to an existing feature. label Jan 8, 2021

freddyaboulton commented Jan 8, 2021

View reviewed changes

freddyaboulton force-pushed the 1580-integrate-ts-classification-into-automl branch from 87ff77b to 0a51e85 Compare January 8, 2021 18:03

freddyaboulton marked this pull request as ready for review January 8, 2021 18:51

freddyaboulton requested review from ParthivNaresh, angela97lin, bchen1116, chukarsten, dsherry and jeremyliweishih January 8, 2021 18:51

chukarsten suggested changes Jan 8, 2021

View reviewed changes

ParthivNaresh approved these changes Jan 10, 2021

View reviewed changes

freddyaboulton force-pushed the 1580-integrate-ts-classification-into-automl branch from 0a51e85 to 8ab890b Compare January 11, 2021 15:17

freddyaboulton requested a review from chukarsten January 11, 2021 15:41

bchen1116 reviewed Jan 12, 2021

View reviewed changes

freddyaboulton mentioned this pull request Jan 12, 2021

Changes and fixes for time series classification problems #1681

Closed

freddyaboulton force-pushed the 1580-integrate-ts-classification-into-automl branch from 8ab890b to 307830e Compare January 12, 2021 19:59

freddyaboulton added 7 commits January 12, 2021 17:08

Integrate ts classification pipelines into automl.

0f93176

Adding PR 1666 to release notes

fa6082d

Renaming TimeSeriesBaselineRegressor to TimeSeriesBaselineEstimator i…

b47c6ef

…n api_ref.

Renaming time_series_baseline_regressor.py to time_series_baseline_es…

7e51ecc

…timator.py

Renaming time series baseline pipeline test file.

93702d9

Adding ts classification baseline pipelines to _not_used_in_automl.

987058c

Deleting test because we have coverage in baseline pipeline tests.

5f1693a

freddyaboulton added 4 commits January 12, 2021 17:08

Reverting bug.

7149adb

Adding coverage for value error in predict_proba.

1a53622

Updating docstings (change regressor to estimator)

e97ce89

condensing expected target value definition in test_time_series_basel…

c636e3b

…ine_pipeline.py

freddyaboulton force-pushed the 1580-integrate-ts-classification-into-automl branch from 307830e to c636e3b Compare January 12, 2021 22:08

Updating regressor -> estimator in time series baseline estimator.

c7fcabe

chukarsten approved these changes Jan 13, 2021

View reviewed changes

freddyaboulton merged commit f49c327 into main Jan 13, 2021

freddyaboulton deleted the 1580-integrate-ts-classification-into-automl branch January 13, 2021 17:50

bchen1116 mentioned this pull request Jan 26, 2021

Release v0.18.0 #1743

Merged

		assert problem_config == {"max_delay": 2, "gap": 3}


		@patch('evalml.pipelines.TimeSeriesRegressionPipeline.score', return_value={"R2": 0.3})

		)


		class TimeSeriesBaselineRegressionPipeline(TimeSeriesRegressionPipeline):

Integrate Time Series Classification Pipelines into AutoMLSearch #1666

Integrate Time Series Classification Pipelines into AutoMLSearch #1666

Uh oh!

Conversation

freddyaboulton commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jan 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton commented Jan 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

freddyaboulton commented Jan 7, 2021 •

edited

Loading

codecov bot commented Jan 7, 2021 •

edited

Loading

freddyaboulton commented Jan 12, 2021 •

edited

Loading