Home

Introduction

Welcome to the SimDesign wiki! This wiki is for front-end users to add their own example Monte Carlo simulations, potentially linking to published articles which have used SimDesign. As well, some additional examples have been included to highlight different types of simulation experiments, and contains a previous presentation file related to the details of simulations and SimDesign. See also the 2020 presentation by Matt Sigal for related presentation information.

If you are interested in a more in-depth description of the package and its use in real-world simulation experiments refer to Chalmers, R. P. & Adkin, M. C. (2020). Writing effective and reliable Monte Carlo simulations with the SimDesign package. The Quantitative Methods for Psychology, 16(4), 248--280 , For a (somewhat dated) discussion of using SimDesign for statistical pedagogy please refer to Sigal, M. J. & Chalmers, R. P. (2016). Play It Again: Teaching Statistics with Monte Carlo Simulation. Journal of Statistics Education, 24, 136-156.

Simulation Examples with SimDesign

• Power Curve Plot - Simple example of how to build empirical power-curve plots.
• Type I Error and Power Rates - Example of how to construct a more realistic Type I error and power analysis simulation. Compares the Welch and Independent t-test when model assumptions are violated. Note that this simulation structure can be reorganized to reflect post-hoc power analyses in empirical research (as well as so-called observed power, though such applications are highly questionable).
• Compromised Power Analysis - Example of how to conduct a compromised power analysis given the trade-off between Type I and Type II error rates when sample size and effect sizes are fixed.
• Power Rates with Unreliable Measurements - Example using paired samples t-tests with unreliable measurements via simple classical test theory results.
• Parameter Recovery - Realistic example of how to compare parameter estimates using different estimators. Compares the mirt and lavaan approaches to item factor analysis and how effectively item parameters are recovered.
• Recovering Secondary Estimates - Example of how to use SimDesign to compute secondary estimates that are of interest (e.g., factor scores and latent trait values). Uses the mirt package, particularly the fscores() function.
• Within-condition Parallelization - Example of how to use SimDesign for computerized adaptive testing when parallel computations are distributed within each replication rather than between. Uses the mirtCAT package.
• Coverage - Simple example of how to determine the coverage rate of an estimator. Example demonstrates the coverage of the t and z-distributions when recovering a population mean.
• Missing Data - Example with MCAR and MNAR missing data mechanisms when recovering correlation coefficients.
• Parametric Bootstrap - Example showing how to use the SimDesign package as a Monte Carlo generation workhorse for parametric bootstrap functions.
• Calling External Binary Programs - Example demonstrating how to utilize Mplus within the SimDesign framework using both single and multi-core estimation strategies.
• Simulation Analyses Using Compiled Stan MCMC Code - Example using Stan's Hamiltonian MCMC to demonstrate how to use multiple compiled Stan files in a Monte Carlo simulation (executed using parallel processing).
• Fractional Factorial Designs - Example demonstrating how to detect/screen factors that may influence a given outcome variable using fractional factorial designs instead of fully-crossed designs.

Into the wild

Example code for simulations which have been previously been published or are visible elsewhere.

• Brown, M. B. and Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364--367.
  • This file contains the simulation code rendered through an Rmarkdown file, organized by design-rows, where not applicable combinations are omitted. The final simulation results are available in this .rds file
• Flora, D. B. & Curran, P. J. (2004). An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis With Ordinal Data. Psychological Methods, 9, 466-491
  • This simulation demonstrates an important result for item factor analysis using a limited information estimation method commonly available in SEM software when the model is slightly mis-specified. Specifically, this study investigated the effects of generating non-normal latent traits and how robust these limited information estimators are to this violation. See this file for the helper functions required for generating syntax with lavaan. The final simulation results are available in this .rds file
• Kang T, Cohen AS. IRT Model Selection Methods for Dichotomous Items. Applied Psychological Measurement. 2007;31(4):331-358. doi:10.1177/0146621606292213
  • This file contains a partial replication of Kang and Cohen's second simulation (partial since the MCMC results were omitted as MCMC methods have evolved better in the more recent years; e.g., computing true-blue Bayes factors via bridge sampling). The results generally replicate well using the mirt software, however selecting a three-parameter logistic model using the likelihood ratio test and AIC criteria was notably better using mirt than when using the older BILOG-MG software. Convenience functions used in this simulation can be found here
• Nordstokke, D. W., & Zumbo B. D. (2007). A Cautionary Tale About Levene's Tests for Equal Variances. Journal of Educational Research and Policy Studies, 7, 1-14.
  • This file reconstructs the Monte Carlo simulation described. Additionally, the Brown-Forsythe approach is included, mostly because R >= SPSS.
• Raju, N. S.; van der Linden, W. J. & Fleer, P. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.
  • This simulation reflects a specific adaption of the Raju et al. (1995) simulation study, but actually performs the replication experiment rather than analyzing particular instances of generated data. It also uses a few modifications, such as estimating the IRT parameters with MML estimation rather than MMAP (with BILOG-MG's defaults). Overall, the results from the simulation were less convincing about the statistical properties of the two-step DFIT statistics, which is in agreement with what Chalmers conjectured in this article.
• Hu, L. & Bentler, P. M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives Structural Equation Modeling, 6, 1-31
  • This simulation presents a re-analysis of Hu and Bentler's (1999) simulation regarding fit statistics in structural equation models under misspecifications. The code also demonstrates why the save_results or store_results flags are useful when post-summarizations via reSummarise() are likely required (e.g., for investigating different rules-of-thumb cut-offs given the simulated information).

For-loop issues

The article written by Hallgren, Kevin A. (2013) “Conducting Simulation Studies in the R Programming Environment.” Tutorials in Quantitative Methods for Psychology, 9 (2). NIH Public Access: 43. explained how to run Monte Carlo simulations in R, but unfortunately introduced the topic using the for loop strategy. So, I decided to compare the code to SimDesign's.

• The result was that, when written in SimDesign's framework, the code was not only cleaner and easier to read, manipulate, and manage, but actually ran about 2-6x faster on a single core. Oddly enough, it also demonstrated some of the pitfalls discussed by Sigal and Chalmers (2016) very nicely, and therefore serves as another good comparison between the coding approaches.