Execution time of multiple forecasts in AutoARIMA in StatsForecast, Python. Index not read correctly?

[anders-kruse]

I want to run +10.000 forecasts on time series using AutoARIMA in Statsforecast. Execution time is super slow when I try to make more than one forecast. I have labelled my time series through the index. It seems Statsforecast dosn't seperate the two time series.

filter_train = df_71["pyth_date"]<='202012'
filter_test  = df_71["pyth_date"]>'202012'

train, test=df_71[filter_train].drop('pyth_dato', axis=1),df_71[filter_test].drop('pyth_dato', axis=1)

display_side_by_side(train.filter(like = '2_2137', axis=0).tail(),
                     test.filter(like = '2_2137', axis=0).head(),
                     train.filter(like = '1_2137', axis=0).tail(),
                     test.filter(like = '1_2137', axis=0).head(),
                     titles=['Train','Test','Train','Test'])

I run the following StatsForecast code:

#forecast modellen
fcst = StatsForecast(
    train,
    models = [(auto_arima,12)],
    freq = 'M',
    n_jobs = 4
)

Y_hat_df_intervals = fcst.forecast(h=12,  level=(85, 95))

print(fcst.uids)

Index(['1_2137', '2_2137'], dtype='object')

Doing a single forecast takes 0.3 seconds. Doing both takes 16 second.

I get the forecast (here as a graph):

How do I get StatsForecast to seperate the index correctly? Looking at the output, it seems like, it sees it as one series.

Thank you very much for looking at it and for sharing your great algorithm!!!

[kdgutier]

Hi @anders-kruse,

Some comments on the code you shared:

1. StatsForecast class is designed to operate with panel data in a pandas DataFrame with 'unique_id', 'ds', and 'y' columns. Here is an example.
2. Seems that your plots have some indexes confused. My intuition is that your indexes are getting combined into their interactions.
3. You might want to set the number of jobs to n_jobs=-1 to use all your local computer's cpus.
4. There is a gap between the first time you run the method and the second, because the first time will compile the numba code.

Can you try to follow the example above and see if that works for you?

[anders-kruse]

Thank you for your swift response!
I'll try to look into the example, see if I can make it work and come back.
It's much appreciated.

[anders-kruse]

hey Kdgutier,
Your answer helped me a lot. It got nothing to do with the index. Using n_job=1 gave me Execution times linear increasing with the number of time series (as it should be, I guess). My problem is (I think), that I have not setup the multiprocessing (numba) right.
I get the following error running n_job=-1:
OSError: [Errno 22] Invalid argument: 'C:\Users\anmk\PycharmProjects\test\'
Traceback (most recent call last):
I will try to look more into that!
Thank you for your patience :)

[kdgutier]

An additional comment in the updated StatsForecast library we are making a distinction between fit, predict methods that store parameters and extra information and its efficient combination forecast that does not store them.

When forecasting large scale data like in your case we recommend the forecast function. If you are interested in studying a particular function we recommend fit and predict.

[anders-kruse]

cool, I was looking for this as well! :)