Principles of forecasting: evaluating the forecast accuracy of a particular point forecast

[nickcox896]

From section 5.8 of Principles of Forecasting link, I modify the code as follows in order to use an arima model:

`recent_production <- aus_production |> filter(year(Quarter) >= 1992)

beer_train <- recent_production |> filter(year(Quarter) <= 2007)

beer_fit <- beer_train |> model(ARIMA(Beer))

beer_fc <- beer_fit |> forecast(h = 4)
beer_fc

1 ARIMA(Beer) 2008 Q1 N(428, 155) 428.
2 ARIMA(Beer) 2008 Q2 N(384, 165) 384.
3 ARIMA(Beer) 2008 Q3 N(400, 165) 400.
4 ARIMA(Beer) 2008 Q4 N(479, 165) 479.
`
accuracy(beer_fc, recent_production)

.model .type ME RMSE MAE MPE MAPE MASE RMSSE ACF1
ARIMA(Beer) Test 4.48 8.43 8.28 1.03 1.94 0.579 0.502 0.0857

So we forecast one year ahead (since we have quartely data) and we obtain the four forecasted values recursively (that is, using the previously forecasted values) and the reported RMSE is calculated by taking the average of the squared prediction errors over the whole forecasting period.

1. Now If I want to obtain the RMSE only for the last forecast period, that is, for 2008 Q4. how can I do that in R?

2. In the above example, I have an ARIMA. Suppose that I also had an ETS. If someone wanted to evaluate the forecasting performace of these two competing models, which of the two RMSEs would someone choose? The one reported by "accuracy" or the one that would use only the last forecast period (2008 Q4)?

3. If someone wanted to see what would happen at the end of one year later( 2008 Q4), which of the two RMSEs would choose? The one reported by "accuracy" or the one that would use only the last forecast period (2008 Q4)?

[robjhyndman]

1. For accuracy from a specific period, you can just filter on that period:

beer_fc |>
  filter(Quarter == yearquarter("2008 Q4")) |>
  accuracy(recent_production) |>
  pull(RMSE)

2/3. You should use accuracy statistics that cover whatever forecast horizons you care about. But I wouldn't use accuracy statistics from a single period, as they can be affected by whatever happened in that particular time period, and do not generalize to other time periods. If you are interested in which model is best at forecasting four steps ahead, then I would use time series cross-validation with a 4 step horizon.

[nickcox896]

Thank you! You have been very helpful with the nice explanations you provide!! Many thanks!