loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)
R
Clone or download
Latest commit ae4714e Apr 24, 2018

README.md

Stan Logo

loo

Travis-CI Build Status codecov CRAN_Status_Badge RStudio_CRAN_mirror_downloads_badge

Efficient approximate leave-one-out cross-validation for fitted Bayesian models

Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps.

The loo R package package implements the fast and stable computations for LOO and WAIC from

  • Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413--1432. doi:10.1007/s11222-016-9696-4. Online, arXiv preprint arXiv:1507.04544.

  • Vehtari, A., Gelman, A., and Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint arXiv:1507.02646.

From existing posterior simulation draws, we compute LOO using Pareto smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing predictive errors between two models.

As of version 2.0.0, the package also provides methods for using stacking and other model weighting techiques to average Bayesian predictive distributions. For details on stacking and model weighting see:

  • Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091. Online, arXiv preprint arXiv:1704.02030.

Resources

Installation

  • Install from CRAN:
install.packages("loo")
  • Install from GitHub (requires devtools package):
if (!require(devtools)) install.packages("devtools")
devtools::install_github("stan-dev/loo")

You can also set build_vignettes=TRUE but the installation is slower and the vignettes are always available online at mc-stan.org/loo/articles/

Python and Matlab/Octave Code

Corresponding Python and Matlab/Octave code can be found at the avehtari/PSIS repository.