[BUG] pmdarima estimators break when X contains more indices than the forecasting horizon

[d4nielmeyer]

Describe the bug
I have a dataset containing 1 endogenous and 5 exogenous variables and would like to do cross-validation (via ExpandingWindowSplitter) to evaluate several AutoARIMA models. In particular, my intention was to use an initial_window = 3, step_length = 1 and fh = [1, 2, 3] (as illustrated below).

| * * * x x x - - - |
| * * * * x x x - - |
| * * * * * x x x - |
| * * * * * * x x x |

X (exogenous variables) is of shape (16, 5), while y (endogenous variable) is of shape (16, ). After running temporal cross-validation I get train and test indices of length 3 for the first split -> train [0, 1, 2] and test [3, 4, 5], which is what I would expect. However, within the temporal CV when I split the data into X/y-train/test I get y_train of shape (3, ), y_test (3, ), X_train (3, 5) but X_test (6, 5). Since X_test: n_rows=6 doesn't match n_periods=3 it is not accepted to be passed inside the predict-method (inside the evaluate-method). As a consequence I get the following error notification:

ValueError('X array dims (n_rows) != n_periods')

To Reproduce

To reproduce the error I used a simple, but similar dataset from the sktime library:

from sktime.datasets import load_longley
from sktime.forecasting.model_evaluation import evaluate
from sktime.forecasting.model_selection import ExpandingWindowSplitter
from sktime.forecasting.arima import AutoARIMA
from sktime.performance_metrics.forecasting import MeanAbsoluteError
import numpy as np

y, X = load_longley()

forecaster = AutoARIMA()
cv = ExpandingWindowSplitter(initial_window=3, step_length=1, fh=np.arange(1, 4))
loss = MeanAbsoluteError()

results = evaluate(forecaster=forecaster, y=y, X=X, cv=cv, error_score='raise', scoring=loss)

Expected behavior
I am pretty aware of the facts that:

• If you fit with exogenous, you must predict with exogenous
• When you are predicting with exogenous, your dimensions must match.

But to some extent I would expect that (S)ARIMAX-models were capable of handling inputs at inference time in a way like:
A)
input:
------------------ y(t-3)
------------------ y(t-2)
------------------ y(t-1)
x1(t) x2(t) x3(t)

output:
y(t)
y(t+1)
y(t+2)
y(t+3)

or B):
input:
x1(t-3) x2(t-3) x3(t-3) y(t-3)
x1(t-2) x2(t-2) x3(t-2) y(t-2)
x1(t-1) x2(t-1) x3(t-1) y(t-1)

output:
y(t)
y(t+1)
y(t+2)
y(t+3)

Versions
System:
python: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00) [Clang 13.0.1 ]
executable: /Users/dmr/miniforge3/envs/explain/bin/python
machine: macOS-12.6-arm64-arm-64bit

Python dependencies:
pip: 22.3
setuptools: 65.4.1
sklearn: 1.1.2
sktime: 0.13.4
statsmodels: 0.13.2
numpy: 1.23.3
scipy: 1.8.1
pandas: 1.4.4
matplotlib: 3.6.0
joblib: 1.2.0
numba: 0.56.2
pmdarima: 1.8.5
tsfresh: 0.17.0

Thanks for any comments/ advice!

[fkiraly]

bug confirmed on python 3.8.12, windows, current main

[fkiraly]

diagnosed the error: the problem is not evaluate, but the pmdarima interfaces which break when X that is larger than the forecasting horizon is passed (which evaluate does).

[fkiraly]

FYI, see #3657 for a discussion of the deeper issue. Comments or suggestions appreciated.

[fkiraly]

I think this is another instance of this bug: #3763