Bayesian bootstrap for uncertainty estimates

[avehtari]

Summary:
Add

• Function for Bayesian bootstrap
• Function for getting BB-samples describing the predictive performance uncertainty

Description:
Bayesian bootstrap can be used to get samples from the distribution of LOO predictive performance estimate. Bayesian bootstrap is equal to Dirichlet distribution model. Function should have optional arguments for alpha and random seed. Alpha not equal to 1, is needed later. Same random seed allows easier comparison of models. See Aki Vehtari and Jouko Lampinen (2002). Bayesian model assessment and comparison using cross-validation predictive densities. Neural Computation, 14(10):2439-2468.

Example code with log score (and seed for Dirichlet not fixed)

data(radon)
y<-radon$log_radon
## Fit the first model
modelA <- stan_lmer(
    log_radon ~ floor + log_uranium + floor:log_uranium + (1 + floor | county),
    data = radon,
    cores = 4,
    iter = 2000,
    chains = 4)
looA<-loo(modelA)
loos<-looA$pointwise[,1]
## number of observations
N<-length(loos)
## number of BB-samples
nb<-10000
## Dirichlet alpha
alpha<-1
## nb samples from the Dirichlet distribution
library(extraDistr)
dirw<-rdirichlet(nb,matrix(alpha,1,N))
## BB-samples from elppd
bbelppd=rowSums(t(t(as.matrix(dirw))*as.vector(loos)))*N

[jgabry]

@avehtari can we close this now that BB is added in loo 2? Or do you want this as something separate/additional?

_{Sent with GitHawk}

[avehtari]

Let's discuss this next week

[ParadaCarleton]

Unsure how much of this was ever implemented -- if it was, it should probably be emphasized more in the vignettes, since I didn't see anything related to it besides a small mention that it can be used as part of model stacking algorithms. (Although I may have missed it!)

If the Bayesian bootstrap has been added, histospline smoothing may be worth looking at for improving inference with small samples. I'm not 100% sure whether BCa can be extended to the Bayesian bootstrap as well, but if so, that might be interesting to add later on.

[avehtari]

Bayesian bootstrap is used by loo_model_weights() which has documentation with examples.

Based on https://arxiv.org/abs/2008.10296 and related additional experiments, it seems the benefit of Bayesian bootstrap in uncertainty estimate is smaller than what I assumed in 2017 when creating this issue. I close this now. If you are interested in histospline, the discussion related to would be better elsewhere or in a new issue.