ENH: information criteria or penalized mean squared error on subsample

[josef-pkt]

just an idea, I'm not sure it will work

Hyndam, e.g. in Hyndman Khandakar (2008) "Automatic Forecasting ..." and in blog post (*) is negative on using information criteria across several model types, e.g. when using different amount of differencing in SARIMA or comparing exponential smoothing with SARIMA, or full MLE versus conditional MLE. One argument is different initial conditions conditions for the process and different sample because some initial observations are dropped.
There is no problem comparing llf based IC for log or box-cox transformation as long as likelihood function is adjusted for the transformation, nor for comparing different distributional assumptions, e.g. normal versus t.)

(*) https://robjhyndman.com/hyndsight/aic/

However, I think we can compute the comparison statistic, ic or penalized mse, just on the last several observations, e.g. a fixed number (appropriate for the frequency and length of the sample data) or a fraction of observations.
This is similar to using a hold-out, test subsample, but it is still in-sample but only on the recent part of the dataset.

One advantage is that it also looks at the subsample that is relevant for forecasting if there are structural breaks, change in trends or seasonal pattern. (related to Chow test or similar)
Compared to out-of-sample forecasting and cross-validation this has still the advantage that it is much faster. (i.e. useful at least for a preliminary search to narrow down which models might be considered for cross-validation.)

Related: measures to compute weights for forecast combination, e.g. EVIEWS documentation summarizes several of them. There seems to be the same dichotomy, either full training sample measures, or out-of-sample forecast measures. In between these, we could use the recent subsample of the training data.

For cross-sectional data this is not very useful because we don't have a prior order of observations. We would have to define some measure for how close the prediction sample (exog) is to subsamples of the training data. (It's a bit like matching literature but matching prediction to training samples.)

related issues should be for automatic forecasting and model selection
e.g. #2571