Calculating measures of forecast accuracy for the test sample using bootstrapping in VAR models: Principles of forecasting

[nickcox896]

I want to calculate the measures of forecast accuracy for the test sample using bootstrapping in VAR models
My code, which seems to work ok, is this one

library(fpp3)
training <- us_change|> filter(year(Quarter) <= 2010)  
h=3
times=5

fc <- training|> model(m1 = VAR(vars(Consumption, Income)~AR(2)),
m2 = VAR(vars(Consumption, Income)~AR(3)))   |>
 forecast(h = h, bootstrap = TRUE, times =times,seed=1)
 
fc |>
  mutate(dist = distributional::dist_degenerate(.mean[,1])) |>
  as_fable(
    response = "Consumption",
    distribution = dist
  ) |>
 forecast:: accuracy(us_change)
 
fc |>
  mutate(dist = distributional::dist_degenerate(.mean[,2])) |>
  as_fable(
    response = "Income",
    distribution = dist
  ) |>
 forecast:: accuracy(us_change)

Is there a simpler code? Furthermore, the above code gives these measures for each equation by model.
Can I get these measures for each model as a whole?

[robjhyndman]

First you can just pass fc to accuracy to get the accuracy statistics by model and variable:

library(fpp3)
training <- us_change |> filter(year(Quarter) <= 2010)
h <- 3
times <- 5
fc <- training |>
  model(
    m1 = VAR(vars(Consumption, Income) ~ AR(2)),
    m2 = VAR(vars(Consumption, Income) ~ AR(3))
  ) |>
  forecast(h = h, bootstrap = TRUE, times = times, seed = 1)
fc |> accuracy(us_change)
#> # A tibble: 4 × 11
#>   .model .response   .type      ME  RMSE   MAE    MPE  MAPE  MASE RMSSE   ACF1
#>   <chr>  <chr>       <chr>   <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>  <dbl>
#> 1 m1     Consumption Test  -0.390  0.408 0.390 -133.   133. 0.568 0.450 -0.555
#> 2 m1     Income      Test  -0.0312 0.549 0.521   78.4  127. 0.560 0.451 -0.258
#> 3 m2     Consumption Test  -0.362  0.475 0.371 -139.   141. 0.540 0.525 -0.504
#> 4 m2     Income      Test  -0.278  0.865 0.853  135.   237. 0.916 0.711 -0.280

As this output is simply a tibble, you can summarise it by group as normal. For example:

fc |>
  accuracy(us_change) |>
  group_by(.model) |>
  summarise(MASE = mean(MASE), RMSSE = sqrt(mean(RMSSE^2)))
#> # A tibble: 2 × 3
#>   .model  MASE RMSSE
#>   <chr>  <dbl> <dbl>
#> 1 m1     0.564 0.450
#> 2 m2     0.728 0.625

Be careful averaging accuracy statistics across variables if they are not on the same scale. MASE and RMSSE are scaled, so there is not problem in averaging them.

[nickcox896]

Perfect! Thank you!!