CRPS() incorrect for sqrt() transformed forecasts

[nickreich]

Here is a reproducible example for what I think is a bug uncovered while developing forecasting evaluation course materials.

I had locally a copy of fabletools 0.8.0 and the website is running a cached version of 0.7.x and we were getting different results for a power-transformed (fourth-root, in this case) forecasts. Below is a minimal reproducible example using fabletools 0.8.0.

Note that at the end the fabletools_CRPS should agree with the sample_truth column and it does for the no transform and log transform but does not agree for the sqrt transform.

## Minimal reproducible example
##
## fabletools::CRPS() over-estimates CRPS for power-transformed
## (e.g. sqrt) forecast distributions.
## It is correct for untransformed (Normal) and log-transformed forecasts.
##
## Ground truth = CRPS of the SAME forecast distribution, estimated from its own
## samples via scoringRules::crps_sample().

suppressMessages({library(fable); library(tsibble); library(dplyr)})

set.seed(1)
n   <- 100
y   <- rnorm(n, mean = 2.5, sd = 0.35)^2      # simple, positive, right-skewed series
dat <- tsibble(t = seq_len(n), y = y, index = t)

obs <- 6                                       # value to score the 1-step forecast against

fit <- dat |> model(
  none = RW(y),          # Normal forecast   (control)
  log  = RW(log(y)),     # log transform     (control)
  sqrt = RW(sqrt(y))     # power transform   (affected)
)
fc <- forecast(fit, h = 1)

res <- lapply(c("none", "log", "sqrt"), function(m) {
  D <- fc |> filter(.model == m) |> pull(y)    # length-1 distribution vector
  set.seed(1)
  data.frame(
    model           = m,
    fabletools_CRPS = fabletools::CRPS(D, obs),
    sample_truth    = scoringRules::crps_sample(obs, distributional::generate(D, 1e6)[[1]])
  )
})
res <- do.call(rbind, res); res$ratio <- res$fabletools_CRPS / res$sample_truth
print(res, digits = 4, row.names = FALSE)
#>  model fabletools_CRPS sample_truth  ratio
#>   none          0.5790       0.5790 0.9999
#>    log          0.5263       0.5258 1.0009
#>   sqrt          2.8696       0.5418 5.2967

sessionInfo()
#> R version 4.4.3 (2025-02-28)
#> Platform: aarch64-apple-darwin20
#> Running under: macOS Sequoia 15.7.7
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] dplyr_1.2.0      tsibble_1.2.0    fable_0.5.0      fabletools_0.8.0
#> 
#> loaded via a namespace (and not attached):
#>  [1] gtable_0.3.6         compiler_4.4.3       vecvec_1.2.0        
#>  [4] tidyselect_1.2.1     reprex_2.1.1         Rcpp_1.1.1-1.1      
#>  [7] dichromat_2.0-0.1    tidyr_1.3.2          scales_1.4.0        
#> [10] yaml_2.3.12          fastmap_1.2.0        ggplot2_4.0.2       
#> [13] R6_2.6.1             generics_0.1.4       distributional_0.5.0
#> [16] knitr_1.51           MASS_7.3-65          ggtime_0.2.0.9000   
#> [19] tibble_3.3.1         mixtime_0.2.0.9000   lubridate_1.9.5     
#> [22] tzdb_0.5.0           RColorBrewer_1.1-3   pillar_1.11.1       
#> [25] scoringRules_1.1.3   rlang_1.2.0          xfun_0.59           
#> [28] S7_0.2.2             fs_2.1.0             otel_0.2.0          
#> [31] timechange_0.4.0     cli_3.6.6            progressr_1.0.0     
#> [34] withr_3.0.3          magrittr_2.0.5       digest_0.6.39       
#> [37] grid_4.4.3           rstudioapi_0.19.0    lifecycle_1.0.5     
#> [40] anytime_0.3.13       vctrs_0.7.1          evaluate_1.0.5      
#> [43] glue_1.8.1           numDeriv_2016.8-1.1  farver_2.1.2        
#> [46] purrr_1.2.0          rmarkdown_2.30       tools_4.4.3         
#> [49] pkgconfig_2.0.3      htmltools_0.5.9

^{Created on 2026-07-16 with reprex v2.1.1}

[mitchelloharawild]

Thanks for sharing the course materials - I had a quick look around - nice work 👍

Your version of {distributional} is quite old, and some improvements to the Taylor series approximations for transformed distributions have been made since (I believe in v0.6.0 this was reworked). I am unable to reproduce this issue with recent versions of the packages, perhaps try updating distributional to see if this fixes it.

## Minimal reproducible example
##
## fabletools::CRPS() over-estimates CRPS for power-transformed
## (e.g. sqrt) forecast distributions.
## It is correct for untransformed (Normal) and log-transformed forecasts.
##
## Ground truth = CRPS of the SAME forecast distribution, estimated from its own
## samples via scoringRules::crps_sample().

suppressMessages({library(fable); library(tsibble); library(dplyr)})
#> Warning: replacing previous import 'fabletools::aggregate_key' by
#> 'tsibble::aggregate_key' when loading 'fable'

set.seed(1)
n   <- 100
y   <- rnorm(n, mean = 2.5, sd = 0.35)^2      # simple, positive, right-skewed series
dat <- tsibble(t = seq_len(n), y = y, index = t)

obs <- 6                                       # value to score the 1-step forecast against

fit <- dat |> model(
  none = RW(y),          # Normal forecast   (control)
  log  = RW(log(y)),     # log transform     (control)
  sqrt = RW(sqrt(y))     # power transform   (affected)
)
fc <- forecast(fit, h = 1)

res <- lapply(c("none", "log", "sqrt"), function(m) {
  D <- fc |> filter(.model == m) |> pull(y)    # length-1 distribution vector
  set.seed(1)
  data.frame(
    model           = m,
    fabletools_CRPS = fabletools::CRPS(D, obs),
    sample_truth    = scoringRules::crps_sample(obs, distributional::generate(D, 1e6)[[1]])
  )
})
res <- do.call(rbind, res); res$ratio <- res$fabletools_CRPS / res$sample_truth
print(res, digits = 4, row.names = FALSE)
#>  model fabletools_CRPS sample_truth  ratio
#>   none          0.5790       0.5790 0.9999
#>    log          0.5263       0.5258 1.0009
#>   sqrt          0.5423       0.5418 1.0009

^{Created on 2026-07-17 with reprex v2.1.1}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.5.2 (2025-10-31)
#>  os       KDE neon User Edition
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Australia/Melbourne
#>  date     2026-07-17
#>  pandoc   3.8.3 @ /usr/share/positron/resources/app/quarto/bin/tools/x86_64/ (via rmarkdown)
#>  quarto   1.8.26 @ /opt/quarto/bin/quarto
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package        * version    date (UTC) lib source
#>  anytime          0.3.13     2026-04-14 [1] RSPM (R 4.5.0)
#>  cli              3.6.6      2026-04-09 [1] CRAN (R 4.5.2)
#>  dichromat        2.0-0.1    2022-05-02 [1] RSPM (R 4.5.0)
#>  digest           0.6.39     2025-11-19 [1] RSPM (R 4.5.0)
#>  distributional   0.8.1.9000 2026-07-17 [1] local
#>  dplyr          * 1.2.1      2026-04-03 [1] RSPM (R 4.5.0)
#>  evaluate         1.0.5      2025-08-27 [1] RSPM (R 4.5.0)
#>  fable          * 0.5.0      2026-01-23 [1] RSPM (R 4.5.0)
#>  fabletools     * 0.7.0.9000 2026-06-27 [1] local
#>  farver           2.1.2      2024-05-13 [1] RSPM (R 4.5.0)
#>  fastmap          1.2.0      2024-05-15 [1] RSPM (R 4.5.0)
#>  fs               2.1.0      2026-04-18 [1] RSPM (R 4.5.0)
#>  generics         0.1.4      2025-05-09 [1] RSPM (R 4.5.0)
#>  ggplot2          4.0.3      2026-04-22 [1] RSPM (R 4.5.0)
#>  ggtime           0.2.0.9000 2026-06-08 [1] local
#>  glue             1.8.1      2026-04-17 [1] RSPM (R 4.5.0)
#>  gtable           0.3.6      2024-10-25 [1] RSPM (R 4.5.0)
#>  htmltools        0.5.9      2025-12-04 [1] RSPM (R 4.5.0)
#>  knitr            1.51       2025-12-20 [1] RSPM (R 4.5.0)
#>  lifecycle        1.0.5      2026-01-08 [1] RSPM (R 4.5.0)
#>  lubridate        1.9.5      2026-02-04 [1] RSPM (R 4.5.0)
#>  magrittr         2.0.5      2026-04-04 [1] RSPM (R 4.5.0)
#>  MASS             7.3-65     2025-02-28 [2] CRAN (R 4.5.2)
#>  mixtime          0.2.0      2026-06-28 [1] local
#>  numDeriv         2016.8-1.1 2019-06-06 [1] RSPM (R 4.5.0)
#>  otel             0.2.0      2025-08-29 [1] RSPM (R 4.5.0)
#>  pillar           1.11.1     2025-09-17 [1] RSPM
#>  pkgconfig        2.0.3      2019-09-22 [1] RSPM (R 4.5.0)
#>  progressr        0.19.0     2026-03-31 [1] RSPM (R 4.5.0)
#>  purrr            1.2.2      2026-04-10 [1] RSPM (R 4.5.0)
#>  R6               2.6.1      2025-02-15 [1] RSPM (R 4.5.0)
#>  RColorBrewer     1.1-3      2022-04-03 [1] RSPM (R 4.5.0)
#>  Rcpp             1.1.1-1.1  2026-04-24 [1] RSPM (R 4.5.0)
#>  reprex           2.1.1      2024-07-06 [1] RSPM (R 4.5.0)
#>  rlang            1.2.0      2026-04-06 [1] CRAN (R 4.5.2)
#>  rmarkdown        2.31       2026-03-26 [1] RSPM (R 4.5.0)
#>  S7               0.2.2      2026-04-22 [1] RSPM (R 4.5.0)
#>  scales           1.4.0      2025-04-24 [1] RSPM (R 4.5.0)
#>  scoringRules     1.1.3      2024-09-18 [1] RSPM (R 4.5.0)
#>  sessioninfo      1.2.4      2026-06-04 [1] CRAN (R 4.5.2)
#>  tibble           3.3.1      2026-01-11 [1] RSPM (R 4.5.0)
#>  tidyr            1.3.2      2025-12-19 [1] RSPM (R 4.5.0)
#>  tidyselect       1.2.1      2024-03-11 [1] RSPM (R 4.5.0)
#>  timechange       0.4.0      2026-01-29 [1] RSPM (R 4.5.0)
#>  tsibble        * 1.2.0.9000 2026-06-08 [1] local
#>  tzdb             0.5.0      2025-03-15 [1] RSPM (R 4.5.0)
#>  vctrs            0.7.3      2026-04-11 [1] RSPM (R 4.5.0)
#>  vecvec           1.2.0.9000 2026-07-06 [1] local
#>  withr            3.0.2      2024-10-28 [1] RSPM (R 4.5.0)
#>  xfun             0.58       2026-06-01 [1] RSPM
#>  yaml             2.3.12     2025-12-10 [1] RSPM (R 4.5.0)
#> 
#>  [1] /home/mitchell/R/x86_64-pc-linux-gnu-library/4.5
#>  [2] /opt/R/4.5.2/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[nickreich]

Thanks for the quick response! Indeed, a local update of distributional changed the results for me as well.