This package can be used to conduct post hoc analyses of resampling results generated by models.
For example, if two models are evaluated with the root mean squared error (RMSE) using 10-fold cross-validation, there are 10 paired statistics. These can be used to make comparisons between models without involving a test set.
There is a rich literature on the analysis of model resampling results such as McLachlan's Discriminant Analysis and Statistical Pattern Recognition and the references therein. This package follows the spirit of Benavoli et al (2017).
tidyposterior uses Bayesian generalized linear models for this purpose and can be considered an upgraded version of the
caret::resamples() function. The package works with rsample objects natively but any results in a data frame can be used.
You can install the released version of tidyposterior from CRAN with:
Install the development version from GitHub with:
# install.packages("devtools") devtools::install_github("tidymodels/tidyposterior")
library(tidyposterior) # See ? precise_example data(precise_example) # Get classification accuracy results for analysis library(dplyr) accuracy <- precise_example %>% select(id, contains("Accuracy")) %>% setNames(tolower(gsub("_Accuracy$", "", names(.)))) accuracy # Model the accuracy results acc_model <- perf_mod(accuracy, seed = 13311, verbose = FALSE) # Extract posterior distributions: accuracy_dists <- tidy(acc_model) # Credible intervals for accuracy per model summary(accuracy_dists)
This project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
For questions and discussions about tidymodels packages, modeling, and machine learning, please post on RStudio Community.
If you think you have encountered a bug, please submit an issue.
Either way, learn how to create and share a reprex (a minimal, reproducible example), to clearly communicate about your code.