IRRsim: An R Package for Simulating Inter-Rater Reliability

This package provides functions and a shiny application to simulate inter-rater reliability statistics based on various scoring and response models. The initial motivation for this package is to understand the relationship between percent rater agreement and intraclass correlation.

Documentation is available at http://irrsim.bryer.org
Slide deck available at https://irrsim.bryer.org/IRRsim-Presentation.html

Install

The development version of IRRsim can be downloaded from Github using the remotes package:

remotes::install_github('jbryer/IRRsim')

Shiny App

Many of features of this package can be explored using the included shiny applications. The IRRsim_shiny() function will run the application.

IRRsim::IRRsim_demo()

IRRsim Shiny App

Data

The analysis done in the vignette requires a large number of simulated datasets. Not only does it take a long time for those datasets to be generated, the file sizes exceed what is allowed in R packages. The data are stored as Github releases. See the data-releases/IRRsimulations.R R script for how these data files are generated. Alternatively, you can download the data using the following commands in R:

download_dir <- getwd()
download.file('https://github.com/jbryer/IRRsim/releases/download/IRRsimData/IRRsimData.rda', 
              paste0(download_dir, '/IRRsimData.rda'))
load(paste0(download_dir, '/IRRsimData.rda))