Add forecast functions to time series regression pipeline #3742

christopherbunn · 2022-10-05T00:53:47Z

Adds get_forecast_periods() and get_forecast_predictions() to time series regression pipelines.

codecov · 2022-10-05T01:00:18Z

Codecov Report

Merging #3742 (ddcbc8c) into main (e57c05a) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3742     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        341     341             
  Lines      35235   35287     +52     
=======================================
+ Hits       35103   35155     +52     
  Misses       132     132

Impacted Files	Coverage Δ
.../pipelines/time_series_classification_pipelines.py	`100.0% <100.0%> (ø)`
evalml/pipelines/time_series_pipeline_base.py	`100.0% <100.0%> (ø)`
...valml/pipelines/time_series_regression_pipeline.py	`100.0% <100.0%> (ø)`
...peline_tests/test_time_series_baseline_pipeline.py	`100.0% <100.0%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

christopherbunn

Changes are going to be only for regression pipelines for now. Can move to base class when we revisit for time series classification problems.

christopherbunn · 2022-10-05T19:16:18Z

evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

+    X, _, y = ts_data(problem_type=ProblemTypes.TIME_SERIES_REGRESSION)
+
+    X_train, y_train = X.iloc[:15], y.iloc[:15]
+    X_validation = X.iloc[15 : (15 + gap + forecast_horizon)]


This holdout dataset has a feature column that is obviously not generated inside of get_forecast_predictions() (i.e. it is multivariate).

christopherbunn · 2022-10-05T19:40:07Z

evalml/pipelines/time_series_regression_pipeline.py

+        )
+
+        # Generate numerical index
+        first_idx = len(X) - 1 if not isinstance(X.index.dtype, int) else X.index[-1]


Using this to create a int index that is sequential after X_train

Why is this so important to have an integer index?

evalml/pipelines/time_series_regression_pipeline.py

evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

eccabay · 2022-10-06T14:20:05Z

evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

+    forecast_preds = clf.get_forecast_predictions(X=X_train, y=y_train)
+    X_val_preds = clf.predict(X_validation, X_train=X_train, y_train=y_train)
+
+    assert_series_equal(forecast_preds, X_val_preds)


Check my understanding here: the only reason these two predictions are the same is because we're using a baseline pipeline, so the extra feature in X doesn't have an impact. For other models though, this isn't going to be the case.

Can we add info/a warning in the function (and docstring, as well as tests) to make users aware that the get_forecast_predictions function only works for univariate problems?

Is that something we're ok with, since we're forecasting?

@eccabay it should work in the multivariate case (when we engineer lagged features) but it doesn't work for KIA features.

These forecasting functions are meant to work for both univariate and multivariate problems. Because we have a forecast horizon and a gap with time series problems, we need to pull from past data. In the case of multivariate problems, we are pulling these features from past rows of X_train after it is transformed through the entire pipeline. For univariate problems, just the periods generated should be fine.

I agree that we should probably use a more complex pipeline that utilizes features though. Updated to include this.

jeremyliweishih

LGTM - good work!

evalml/pipelines/time_series_regression_pipeline.py

evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

jeremyliweishih · 2022-10-06T17:59:52Z

evalml/tests/pipeline_tests/test_time_series_baseline_pipeline.py

+    forecast_preds = clf.get_forecast_predictions(X=X_train, y=y_train)
+    X_val_preds = clf.predict(X_validation, X_train=X_train, y_train=y_train)
+
+    assert_series_equal(forecast_preds, X_val_preds)


@eccabay it should work in the multivariate case (when we engineer lagged features) but it doesn't work for KIA features.

eccabay

Looks great, thanks Chris!

eccabay · 2022-10-07T12:47:31Z

evalml/pipelines/time_series_regression_pipeline.py

+            ...                                                     "pipeline": {"gap": gap, "max_delay": 1, "forecast_horizon": forecast_horizon, "time_index": "date"}},
+            ...                                        )
+            >>> pipeline.fit(X, y)
+            pipeline = TimeSeriesRegressionPipeline(component_graph={'Linear Regressor': ['Linear Regressor', 'X', 'y']}, parameters={'Linear Regressor':{'fit_intercept': True, 'normalize': True, 'n_jobs': -1}, 'pipeline':{'gap': 1, 'max_delay': 1, 'forecast_horizon': 2, 'time_index': 'date'}}, random_seed=0)


I don't think we need this line

chukarsten · 2022-10-07T16:05:10Z

evalml/pipelines/time_series_regression_pipeline.py

+        )
+
+        # Generate numerical index
+        first_idx = len(X) - 1 if not isinstance(X.index.dtype, int) else X.index[-1]


Why is this so important to have an integer index?

christopherbunn force-pushed the TML-5072_predict_on_full_data branch from 69392fc to 1ca503c Compare October 5, 2022 00:53

christopherbunn changed the title ~~Add function to forecast on time series regression pipeline~~ Add ability to forecast on time series regression pipeline Oct 5, 2022

christopherbunn changed the title ~~Add ability to forecast on time series regression pipeline~~ Add forecast functions to time series regression pipeline Oct 5, 2022

christopherbunn force-pushed the TML-5072_predict_on_full_data branch from 3f4d619 to 2f2a53d Compare October 5, 2022 19:10

christopherbunn commented Oct 5, 2022

View reviewed changes

christopherbunn marked this pull request as ready for review October 5, 2022 19:49

auto-assign bot assigned christopherbunn Oct 5, 2022

christopherbunn requested review from jeremyliweishih, eccabay, chukarsten, dvreed77, fjlanasa and ParthivNaresh October 5, 2022 19:50

eccabay reviewed Oct 6, 2022

View reviewed changes

christopherbunn force-pushed the TML-5072_predict_on_full_data branch from 2d663a4 to 0816cb6 Compare October 6, 2022 19:43

christopherbunn added 5 commits October 6, 2022 15:49

Updated implementation

3105684

Added test cases

9faadf9

Release notes and docstring

dab0a1c

Updated test

cace474

Updated docstring

1949ba2

christopherbunn force-pushed the TML-5072_predict_on_full_data branch from 0816cb6 to 1949ba2 Compare October 6, 2022 19:50

christopherbunn requested a review from eccabay October 6, 2022 19:53

jeremyliweishih approved these changes Oct 6, 2022

View reviewed changes

christopherbunn added 2 commits October 6, 2022 16:50

Fixed docstring

dd98a20

Updated test

717d557

eccabay approved these changes Oct 7, 2022

View reviewed changes

Typo in docstring

ddcbc8c

chukarsten approved these changes Oct 7, 2022

View reviewed changes

chukarsten enabled auto-merge (squash) October 7, 2022 16:09

chukarsten merged commit 77459a6 into main Oct 7, 2022

chukarsten deleted the TML-5072_predict_on_full_data branch October 7, 2022 16:17

ivanayx mentioned this pull request Oct 20, 2022

Release v0.60.0 #3762

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add forecast functions to time series regression pipeline #3742

Add forecast functions to time series regression pipeline #3742

christopherbunn commented Oct 5, 2022 •

edited

Loading

codecov bot commented Oct 5, 2022 •

edited

Loading

christopherbunn left a comment

christopherbunn Oct 5, 2022

christopherbunn Oct 5, 2022

chukarsten Oct 7, 2022

eccabay Oct 6, 2022

jeremyliweishih Oct 6, 2022

christopherbunn Oct 6, 2022

jeremyliweishih left a comment

jeremyliweishih Oct 6, 2022

eccabay left a comment

eccabay Oct 7, 2022

chukarsten Oct 7, 2022

Add forecast functions to time series regression pipeline #3742

Add forecast functions to time series regression pipeline #3742

Conversation

christopherbunn commented Oct 5, 2022 • edited Loading

codecov bot commented Oct 5, 2022 • edited Loading

Codecov Report

christopherbunn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eccabay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christopherbunn commented Oct 5, 2022 •

edited

Loading

codecov bot commented Oct 5, 2022 •

edited

Loading