[timeseries] Add wrappers for Statsforecast models

[shchur]

Description of changes:

• Add AutoETS, AutoARIMA and DynamicOptimizedTheta models from StatsForecast.

To Do:

• Add tests
• Benchmark & add models to presets
• Expose the n_jobs parameter

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Job PR-2758-7212948 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2758/7212948/index.html

[github-actions]

Job PR-2758-a376e62 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2758/a376e62/index.html

[tonyhoo]

Are we planning to add them to any popular presets such as medium_quality or high_quality?

[tonyhoo]

Can we make 1.4.0 the lower bound?

[shchur]

This way we are protecting ourselves from potentially breaking changes / regressions caused by newer versions of the dependencies. Both things already happened to us a few times (caused by minor releases of sktime & GluonTS), so I would rather be extra cautious here.

[tonyhoo]

The fit time should be dependent on instance type as well. If time_limit is not implemented and we expect the local model fit shall be quick compared with other models, shall we send warning message instead?

[shchur]

Updated this logic based on the discussion below.

[tonyhoo]

One quick idea to achieve time_limit is to take advantage of timeout from ThreadPoolExecutor or ProcessPoolExecutor. Details can be found here

[shchur]

I agree but, if I understand correctly, this would require monkey-patching StatsForecast. Is it fine if we leave it as-is for now and add a TODO comment?

[tonyhoo]

Let's add it in TODO and follow up with that

[tonyhoo]

How do we determine the priorities of these stats models? How will they added to existing ARIMA and ETA impl? Some accuracy metrics comparison might be helpful to give insights with ensemble results

[shchur]

I set the priorities inversely proportional to the average time it takes to fit these models (slower model -> lower priority).

[shchur]

Also added models to the presets based on the discussion below.

[shchur]

After benchmarking AutoETS, AutoARIMA and DynamicOptimizedTheta models on 28 datasets on m5.4xlarge:

• Fit time (i.e., prediction time for the validation set)
  • ≥25s for all datasets, median time is 45s (these are approximately the same for all models)
  • For 90% of the datasets these times are slower than the respective Statsmodels models (ETS, ARIMA, Theta) by 30s in the median case
  • The runtime numbers are similar on my p3.8xlarge cloud desktop
  • 99% of the time is spent inside StatsForecast.forecast, so our wrapper does not introduce any noticeable overhead
• Performance comparison (MASE) on the test set
  • AutoETS > ETS winrate = 71%
  • AutoARIMA > ARIMA winrate = 62%
  • DynamicOptimizedTheta > Theta winrate = 57%

Based on these findings, I suggest to

• Add AutoETS to the medium_quality preset
• Add AutoETS and AutoARIMA to the high_quality & best_quality presets + remove HPO for ARIMA, ETS in these presets.
• Add DynamicOptimizedTheta to the best_quality preset only.
• Raise TimeLimitExceeded if less than 10s remaining, warn that the model might exceed the time limit if less than 30s remaining.

I will do another round of benchmarking after adding TFT & updating the SFF model from the new GluonTS release. We can update the presets based on these results.

[github-actions]

Job PR-2758-ed4706a is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2758/ed4706a/index.html