Skip to content
loo R package for approximate leave-one-out cross-validation (LOO-CV) and Pareto smoothed importance sampling (PSIS)
R
Branch: master
Clone or download
jgabry Merge pull request #118 from mcol/issue_117
Don't issue a warning each time a waic object is printed (closes #117)
Latest commit 45fd203 Sep 20, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
R Don't issue a warning each time a waic object is printed (#117) Aug 9, 2019
data-raw Generate skewed and fat tailed posterior Dec 18, 2018
data Include datasets with package instead of as csv Apr 4, 2018
inst a bunch of minor cleaning May 13, 2019
man-roxygen Added large data references Jul 1, 2019
man Small fix and comment out failing test case Aug 7, 2019
tests Don't issue a warning each time a waic object is printed (#117) Aug 9, 2019
vignettes Fix generation of vignettes Aug 8, 2019
.Rbuildignore update .gitignore, .Rbuildignore Mar 12, 2019
.gitignore update .gitignore, .Rbuildignore Mar 12, 2019
.travis.yml And finally, add covr Aug 7, 2019
DESCRIPTION Bump version Aug 7, 2019
LICENSE
NAMESPACE Remove nparameters Aug 6, 2019
NEWS.md add tests for loo_compare warnings May 10, 2019
README.md Update README.md May 10, 2019
loo.Rproj Update loo_expectation Mar 18, 2017

README.md

Stan Logo

loo

Travis-CI Build Status codecov CRAN_Status_Badge RStudio_CRAN_mirror_downloads_badge

Efficient approximate leave-one-out cross-validation for fitted Bayesian models

loo is an R package that allows users to compute efficient approximate leave-one-out cross-validation for fitted Bayesian models, as well as model weights that can be used to average predictive distributions.

Leave-one-out cross-validation (LOO-CV, or LOO for short) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps.

The loo R package package implements the fast and stable computations for approximate LOO-CV and WAIC from

  • Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413--1432. doi:10.1007/s11222-016-9696-4. Online, arXiv preprint arXiv:1507.04544.

  • Vehtari, A., Gelman, A., and Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint arXiv:1507.02646.

From existing posterior simulation draws, we compute approximate LOO-CV using Pareto smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing predictive errors between two models.

We recommend PSIS-LOO-CV instead of WAIC, because PSIS provides useful diagnostics and effective sample size and Monte Carlo standard error estimates.

As of version 2.0.0, the loo package also provides methods for using stacking and other model weighting techiques to average Bayesian predictive distributions. For details on stacking and model weighting see:

  • Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091. Online, arXiv preprint arXiv:1704.02030.

Resources

Installation

  • Install from CRAN:
install.packages("loo")
  • Install from GitHub (requires devtools package):
if (!require(devtools)) install.packages("devtools")
devtools::install_github("stan-dev/loo")

We do not recommend setting build_vignettes=TRUE when installing from GitHub because the vignettes take a long time to build and are always available online at mc-stan.org/loo/articles/.

Python and Matlab/Octave Code

Corresponding Python and Matlab/Octave code can be found at the avehtari/PSIS repository.

You can’t perform that action at this time.