Validation repo

This repo contains code and results that validate the power results from the PUMP package.

This package compares the PUMP package power estimation with a simulation-based approach to validate that the power calculations are correct.

Development of this project was supported by a grant from the Institute of Education Sciences (R305D170030).

Authors:

• Kristen Hunter
• Luke Miratrix
• Kristin Porter
• Zarni Htet

Validation results

The complete set of validation results can be found in validation/results. This folder contains Rmarkdown files and output PDF files for all validation scenarios.

A written summary and summary tables of the results can be be found at validation/summary.pdf.

For more detailed results for a particular scenario, reference the document for that scenario. Each file shows graphs comparing power estimated using monte carlo simulations compared to PUMP and PowerUP. Note that PowerUp only works with a single outcome, M = 1. For a general guide to the format of these documents, see validatate_format.

The script we used for running all validation scenarios is validation/code/gen_validation_results_all.R. This script can take several days to run, depending on choice of simulation parameters.

Running your own validation

To run your own validation of a specific scenario, use validation/code/gen_validation_results_one.R.
Set the design, model, and parameters to your desired values. For more information about these choices, see the PUMP package documentation.

You can choose what ways to calculate power to compare calculations:

• runSim runs Monte Carlo simulations to calculate power
• runPump runs the PUMP package to calculate power
• runPowerUp runs the Powerup package to calculate power.

You can run validate_power(), validate_mdes(), or validate_sample(). For validate_sample(), you can choose the appropriate sample type, J, K, or nbar depending on the design.

The functions return data frames with estimates and lower and upper confidence interval bounds (for monte carlo simulations) of the requested parameter.

Folder structure

• code
  • adjust_WY.R: Code specific to Westfall-young adjustments
  • estimate_power_with_simulation.R: Estimating power with Monte Carlo Simulations
  • gen_validation_results_all.R: Script that runs all validation scenarios
  • gen_validation_results_one.R: Script that runs a single validation scenario
  • misc.R: supporting functions
  • validate_power.R: wrapper functions that calculate power by simulation, pump, powerup
  • cloud: contains code for running batch scripts in the cloud
    • readme.txt: reminder for how to run all validation scenarios on the cloud
    • run_sim.txt: bash script to run all validation scenarios in parallel on the cloud
    • validate_sim_parallel.slurm: slurm file for running all simulations in parallel on the cloud
    • validate_pum.slurm: slurm file for running PUM and PUP results on the cloud
• results
  • contains markdown documents and rendered PDFs of all validation scenarios.
  • for a general guide to the format of these documents, see validate_format.