Opportunity for simulation-based calibration

[wlandau]

I think it might be possible to compute the rank statistics for simulation-based calibration using summarize_draws(). Here is a sketch that gets at the idea:

library(posterior)
prior_predictive_draw <- list(
  mu = 4,
  tau = 4,
  theta = c(6, 5, 3, 5, 3, 4, 6.5, 4.5)
)
rank_statistic <- function(draws, prior) {
  # ...
}
summarize_draws(
  example_draws("eight_schools"),
  ranks = ~rank_statistic(.x, prior = prior_predictive_draw)
)

Is there a way for summarize_draws() to tell rank_statistic() which variable it is currently processing? If that were possible, I could easily make stantargets set up large scalable HPC-backed SBC studies, as opposed to the naive interval-based validation method described here.

[paul-buerkner]

Currently, one would have to subset the draws object, via subset_draws for each parameter, which I agree is not ideal. Suggestions on how to support such a feature are very welcome. I will also think about it in more detail.

[paul-buerkner]

@martinmodrak, in light of the existing of the SBC package that already uses posterior I think, how relevant would be this issue here still?

[martinmodrak]

I'd like to think that SBC package will serve everybody well, but maybe there are people who don't want the extra dependency and also want more control?

In any case the function calculate_ranks_draws_matrix (docs: https://hyunjimoon.github.io/SBC/reference/calculate_ranks_draws_matrix.html, code: https://github.com/hyunjimoon/SBC/blob/master/R/results.R#L1022) should let you compute that quite efficiently in case you need to setup different workflow than what the SBC package supports.

[paul-buerkner]

Thank you for your quick feedback! Closing this issue.