Tutorial on model assessment, model selection and inference after model selection
Switch branches/tags
Nothing to show
Clone or download
Latest commit 10296d1 Nov 14, 2018
Permalink
Failed to load latest commit information.
.gitignore update to projpred 1.0 and other updates Oct 10, 2018
README.md updated README.md Nov 14, 2018
Vehtari_StanHEL_CV.pdf added StanCon 2018 Helsinki slides Sep 11, 2018
betablockers.Rmd updated Oct 10, 2018
betablockers.html updated Oct 10, 2018
binom_odds_comparison.R Initial commit Jan 10, 2018
binom_test_densities.RData Initial commit Jan 10, 2018
bodyfat.Rmd fix Oct 10, 2018
bodyfat.html fix Oct 10, 2018
bodyfat.txt stability of projpred - bodyfat example Mar 6, 2018
bodyfat_bootstrap.R update to projpred 1.0 and other updates Oct 10, 2018
bodyfat_bootstrap.RData update to projpred 1.0 and other updates Oct 10, 2018
bodyfat_kfoldcv.R fixed bodyfat and added it back to readme Mar 30, 2018
bodyfat_kfoldcv.RData update to projpred 1.0 and other updates Oct 10, 2018
bodyfat_kfoldcv2.R update to projpred 1.0 and other updates Oct 10, 2018
bodyfat_kfoldcv2.RData update to projpred 1.0 and other updates Oct 10, 2018
candy.Rmd update to projpred 1.0 and other updates Oct 10, 2018
candy.html update to projpred 1.0 and other updates Oct 10, 2018
collinear.Rmd typo fix Oct 10, 2018
collinear.html typo fix Oct 10, 2018
cv_basics.R added cross-validation basics Sep 11, 2018
cv_basics.html added cross-validation basics Sep 11, 2018
diabetes.Rmd some updates to diabetes Nov 5, 2018
diabetes.csv Initial commit Jan 10, 2018
diabetes.html some updates to diabetes Nov 5, 2018
harvard-cite-them-right.csl don't et al with less than 7 names in references Oct 10, 2018
mesquite.Rmd fixed typo Oct 10, 2018
mesquite.dat new mesquite demo Jan 16, 2018
mesquite.html fixed typo Oct 10, 2018
modelsel.bib update to projpred 1.0 and other updates Oct 10, 2018
modelselection_tutorial_slides.pdf added BA online link to Stacking paper Jan 17, 2018
projpredpct.R stability of projpred - bodyfat example Mar 6, 2018
regularizedhorseshoe_slides.pdf regularized horseshoe Jan 12, 2018
roaches.Rmd update to projpred 1.0 and other updates Oct 10, 2018
roaches.html update to projpred 1.0 and other updates Oct 10, 2018
slides_model_assesment_selection.pdf added Masterclass slides Nov 14, 2018
winequality-red.Rmd update to projpred 1.0 and other updates Oct 10, 2018
winequality-red.csv add red wine quality data Feb 27, 2018
winequality-red.html update to projpred 1.0 and other updates Oct 10, 2018

README.md

Model assesment, selection and inference after selection

Example notebooks in R using rstanarm, rstan, bayesplot, loo, projpred.

Talks

  • Model assessment, comparison and selection at Master class in Bayesian statistics, CIRM, Marseille
  • Model assessmen and model selection aka Basics of cross-validation tutorial at StanCon 2018 Helsinki
  • Regularized horseshoe talk at StanCon 2018 Asilomar
  • Model selection tutorial at StanCon 2018 Asilomar

Outline of the StanCon 2018 Asilomar tutorial and links to notebooks

  • Basics of predictive performance estimation
  • When cross-validation is not needed
  • When cross-validation is useful
    • We don't trust the model - roaches
    • Complex model with posterior dependencies - collinear
  • On accuracy of cross-validation
  • Cross-validation and hierarchical models
  • When cross-validation is not enough
  • loo 2.0
  • Projection predictive model selection

Additional case studies

See also

References

  • Bürkner, P.-C., Gabry, J., Vehtari, A. (2018). Leave-one-out cross-validation for non-factorizable normal models. arXiv:1810.10559
  • Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997–1016. Preprint
  • Piironen, J., Paasiniemi, M., and Vehtari, A. (2018). Projective Inference in High-dimensional Problems: Prediction and Feature Selection. arXiv:1810.02406
  • Piironen, J. and Vehtari, A. (2016), Comparison of Bayesian predictive methods for model selection, Statistics and Computing 27(3), 711–735. Online
  • Piironen, J., and Vehtari, A. (2017). On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:905-913. Online
  • Piironen, J., and Vehtari, A. (2017). Sparsity information and regularization in the horseshoe and other shrinkage priors. In Electronic Journal of Statistics, 11(2):5018-5051. Online
  • Piironen, J., and Vehtari, A. (2018). Iterative supervised principal components. Proceedings of the 21th International Conference on Artificial Intelligence and Statistics, accepted for publication. arXiv preprint arXiv:1710.06229
  • Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5):1413–1432. arXiv preprint.
  • Vehtari, A., Gelman, A., Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint.
  • Vehtari, A., Mononen, T., Tolvanen, V., and Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models. JMLR, 17(103):1–38. Online
  • Vehtari, A. and Ojanen, J.: 2012, A survey of Bayesian predictive methods for model assessment, selection and comparison, Statistics Surveys 6, 142–228. Online
  • Williams, D. R., Piironen, J., Vehtari, A., and Rast, P. (2018). Bayesian estimation of Gaussian graphical models with projection predictive selection. arXiv:1801.05725
  • Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2017). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091, Online