Relax contraints on test set for time series

[freddyaboulton]

Pull Request Description

Fixes #3070

---

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

[codecov]

Codecov Report

Merging #3071 (76fe47e) into main (292d5aa) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3071     +/-   ##
=======================================
+ Coverage   99.8%   99.8%   +0.1%     
=======================================
  Files        312     312             
  Lines      30421   30433     +12     
=======================================
+ Hits       30330   30343     +13     
+ Misses        91      90      -1     

Impacted Files	Coverage Δ
.../pipelines/time_series_classification_pipelines.py	100.0% <100.0%> (ø)
evalml/pipelines/time_series_pipeline_base.py	100.0% <100.0%> (+1.0%)	⬆️
.../tests/pipeline_tests/test_time_series_pipeline.py	99.8% <100.0%> (+0.1%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 292d5aa...76fe47e. Read the comment docs.

[ParthivNaresh]

Solid work mate, just left a question about our implementation of _move_index_forward. I think it might be difficult for OS users to understand the limitations of what they can do with forecasting time series when the test set is more than gap observations beyond the training set.

[ParthivNaresh]

"the test set size must be less than..."

[ParthivNaresh]

I like this implementation but I had a question about it. Previously if a user were to set a forecast for indices that were more than gap units ahead of the training data, the second part of the warning above would be relevant: its index needs to start {self.gap + 1} values ahead of the training index.

Now the same forecast attempt doesn't raise the error, but still sets the forecasts to the end of the training set.

This is really useful for when the test set starts at 0 but if it continues the index from the end of the training set, this might give a false impression of future predictions. Maybe we should add a check here for test indices that begin more than gap observations away from the training set so the original error can still be raised.

[freddyaboulton]

Great question @ParthivNaresh ! I think the reason why I didn't want the check to be based on the index is that the index will probably not start at gap + 1 units away from the training/validation index.

My reasoning is that the kind of examples we've been working off, where we manually split the same dataset into separate chunks are not reflective of what happens in "real life". Users will give one dataset to AutoMLSearch (that will have index from 0 to X.shape[0] or whatever) and then the test set will come from a different csv or file. When you read that in with pandas the index will start at 0. I feel like it's awkward for users to have to re-index when we can do that calculation for them.

That being said, it does assume the user is uploading data that's temporally right after the dataset given to AutoMLSearch. We can check that via the new date_index parameter that we're now requiring!

I was actually thinking of doing that in a separate PR but the implementation should be small enough to do in this one maybe?

[ParthivNaresh]

I absolutely agree with the intent behind this. Users are almost guaranteed in the vast majority of cases to pass in a separate dataset for testing and forecasting from index 0, so this works great for that.

In the case that a user passes in a test set separately but tries to do something similar to above, where they might try and forecast values starting from [50 : ] as opposed to starting with the first observation, they'll run into a similar issue. But if that's going in as part of a separate PR I'm totally good with leaving it out of this one!