Comparing VAR and ARIMA: Principles of forecasting

[nickcox896]

Suppose we want to compare the forecasts from two models (VAR and ARIMA). For example,

library(fpp3)
h <- 10
training <- us_change |> filter(year(Quarter) <= 2010)

# The VAR model

fc1 <-  training |>
  model(m1 = VAR(vars(Consumption, Income)))|>
  forecast(h = h) 

measures1<- fc1 |> accuracy(us_change)

## A tibble: 2 × 11
  .model .response   .type     ME  RMSE   MAE    MPE  MAPE  MASE RMSSE   ACF1
  <chr>  <chr>       <chr>  <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>  <dbl>
1 m1     Consumption Test  -0.415 0.467 0.426 -256.   257. 0.621 0.515 -0.440
2 m1     Income      Test  -0.415 1.77  1.12    70.5  105. 1.20  1.45  -0.511


measures11<- measures1 |>
  group_by(.model) |>
  summarise(MASE = mean(MASE), RMSSE = sqrt(mean(RMSSE^2)))

# A tibble: 1 × 3
  .model  MASE RMSSE
  <chr>  <dbl> <dbl>
1 m1     0.910  1.09


## The ARIMA model
 
fc2 <-  training  |>
  model(m3=ARIMA(Consumption)) |>
  forecast(h = h)

measures2<- fc2 |> accuracy(us_change)

 A tibble: 1 × 10
  .model .type     ME  RMSE   MAE   MPE  MAPE  MASE RMSSE   ACF1
  <chr>  <chr>  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>  <dbl>
1 m3     Test  -0.428 0.481 0.432 -261.  262. 0.629 0.531 -0.563

Q1:
Suppose we want to compare the forecast accuracy of the two models. In this case, we should compare the RMSSE from measures2 (0.531) with the RMSSE from measures11 (1.09) ? Is that correct?

Q2:
Suppose we want to compare the forecast accuracy of the two models in terms of the Consumption variable. In this case, we should use the RMSE from measures2 (0.481) with the RMSE of m1 model from measures1 (0.467). Is that correct?

[robjhyndman]

You can only compare the forecast accuracy with respect to the Consumption variable, since that is the only variable common to both models. For that variable, you can compare any of the corresponding measures.

[nickcox896]

THank you!