Skip to content
Bayesian comparisons of models using resampled statistics
R CSS
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Issue template Nov 16, 2017
R commented out a few more lines Nov 15, 2018
data
docs Merge branch 'master' into fix/vignette-error Feb 25, 2019
inst/examples
man commented out a few more lines Nov 15, 2018
pkgdown css changes for pkgdown Nov 18, 2017
revdep documentation updates for next release Nov 15, 2018
tests removed file checks Nov 15, 2018
vignettes update dev docs Feb 25, 2019
.Rbuildignore access data remotely Nov 15, 2018
.gitignore documentation updates for next release Nov 15, 2018
.travis.yml
DESCRIPTION version bump Feb 25, 2019
NAMESPACE export a more explict tidy method so that the generics docs pick it up Nov 15, 2018
NEWS.md
README.md fixed repo name Sep 12, 2018
_pkgdown.yml
codecov.yml Setup for travis and covr Oct 25, 2017
tidyposterior.Rproj
tidyposterior_hex.png Hex logo Dec 16, 2017

README.md

Travis build status Coverage status CRAN_Status_Badge Downloads

This package can be used to conduct post hoc analyses of resampling results generated by models.

For example, if two models are evaluated with the root mean squared error (RMSE) using 10-fold cross-validation, there are 10 paired statistics. These can be used to make comparisons between models without involving a test set.

There is a rich literature on the analysis of model resampling results such as McLachlan's Discriminant Analysis and Statistical Pattern Recognition and the references therein. This package follows the spirit of Benavoli et al (2017).

tidyposterior uses Bayesian generalized linear models for this purpose and can be considered an upgraded version of the caret::resamples function. The package works with rsample objects natively but any results in a data frame can be used.

Installation

You can install tidyposterior from github with:

# install.packages("devtools")
devtools::install_github("tidymodels/tidyposterior")

Example

library(tidyposterior)
# See ? precise_example
data(precise_example)

# Get classification accuracy results for analysis

library(dplyr)
accuracy <- precise_example %>%
   select(id, contains("Accuracy")) %>%
   setNames(tolower(gsub("_Accuracy$", "", names(.)))) 
accuracy

# Model the accuracy results
acc_model <- perf_mod(accuracy, seed = 13311, verbose = FALSE)   

# Extract posterior distributions:
accuracy_dists <- tidy(acc_model)

# Credible intervals for accuracy per model
summary(accuracy_dists)
You can’t perform that action at this time.