Add more tests for score()

[nikosbosse]

It would be good to add a few more tests for score().

Sample case:
What happens if you only have a single sample?

Quantile case:
what happens if you only

• submit the median
• a single quantile that is not the median
• an asymmetric interval

[seabbs]

Do we have any progress on this? Maybe we could update to a checklist to make this more modular

[seabbs]

Anyone keen to chip in here on some infra (maybe @jamesmbaazam :))

[nikosbosse]

In the process of adding some tests.

The following leads to a lot of warnings which need to be cleaned up eventually. Some of them are fine and expected, others like the "Column 'dispersion' does not exist to remove" or "! Function execution failed, returning NULL. Error: argument is of length zero." are a bit opaque.

# only one quantile that is not the median 
  onlyonequantile <- example_quantile[quantile_level == 0.3] %>%
    as_forecast()

  expect_error(
    score(onlyonequantile, metrics = metrics_quantile(
      exclude = c("interval_coverage_50", "interval_coverage_90")
    )),
    ""
  )
  expect_warning(
    score(onlyonequantile, metrics = metrics_quantile(
      exclude = c("interval_coverage_50", "interval_coverage_90")
    )),
    "Function execution failed, returning NULL. Error: object 'upper' not found."
  )

1: ! Function execution failed, returning NULL. Error: object 'upper' not found.
2: In `[.data.table`(forecast, , `:=`((metric_name), do.call(run_safely,  ... :
  Column 'wis' does not exist to remove
3: ! Function execution failed, returning NULL. Error: object 'upper' not found.
4: In `[.data.table`(forecast, , `:=`((metric_name), do.call(run_safely,  ... :
  Column 'overprediction' does not exist to remove
5: ! Function execution failed, returning NULL. Error: object 'upper' not found.
6: In `[.data.table`(forecast, , `:=`((metric_name), do.call(run_safely,  ... :
  Column 'underprediction' does not exist to remove
7: ! Function execution failed, returning NULL. Error: object 'upper' not found.
8: In `[.data.table`(forecast, , `:=`((metric_name), do.call(run_safely,  ... :
  Column 'dispersion' does not exist to remove
9: In min(quantile_level[quantile_level > 0.5]) :
  no non-missing arguments to min; returning Inf
10: ! Function execution failed, returning NULL. Error: argument is of length zero.
11: In `[.data.table`(forecast, , `:=`((metric_name), do.call(run_safely,  ... :
  Column 'bias' does not exist to remove
12: ✖ To compute interval coverage deviation, all quantiles must form central symmetric prediction intervals.
ℹ Missing quantiles: 0.7. Returning NA.
13: ✖ In order to compute the absolute error of the median, "0.5" must be among the quantiles given.
ℹ Returning "NA".

Similar for the case of one asymmetric quantile:

 # one asymmetric interval
  oneasymmetric <- example_quantile[quantile_level %in% c(0.2, 0.6)] %>%
    as_forecast()

  expect_warning(
    expect_warning(
      suppressMessages(
        score(onlyonequantile, metrics = metrics_quantile(
          exclude = c("interval_coverage_50", "interval_coverage_90")
        ))
      ),
      "To compute interval coverage deviation, all quantiles must form central symmetric prediction intervals."
    ),
    'In order to compute the absolute error of the median'
  )

[nikosbosse]

I spun up new issues, #800 and #801, that came up when trying to address this. Once these are addressed I think it makes sense to circle back and include the two tests I mentioned in the last comment.

[nikosbosse]

The "Column 'dispersion' does not exist to remove" errors are addressed in #801.

[jamesmbaazam]

Anyone keen to chip in here on some infra (maybe @jamesmbaazam :))

Sorry, I missed this notification.

[nikosbosse]

Update: some things disappeared, but this has still a few errors/warnings:

onlyonequantile <- example_quantile[quantile_level == 0.3] |>
  as_forecast()

score(onlyonequantile, metrics = metrics_quantile(
  exclude = c("interval_coverage_50", "interval_coverage_90")
))

Working on them :)