This Github repository has the fully reproducible simulation code for:
Carter, E. C., Schönbrodt, F. D., Hilgard, J., & Gervais, W. (2018). Correcting for bias in psychology: A comparison of meta-analytic methods. Retrieved from https://osf.io/rf3ys/.
More material can be found in the associated OSF project.
General workflow for reproducing the analyses
- 0-start.R loads all necessary packages and sources several files. This file should always be loaded first.
- 1-simFramework.R simulates all raw data files /(i.e., the simulated primary studies) and stores them in /simParts. Each of the 432 conditions gets its own file with 1000 simulated meta-analyses. The most recent revision is stored in simPartsRev2
- 2-analysisFramework.R grabs all simulated files from /simParts, runs all meta-analytical techniques on them, and stores the results as separate files under /analysisParts. The most recent revision is stored in analysisPartsRev2
- 3-resultsFramework.R loads the results of the meta-analyses from /analysisPartsRev2, aggregates them in a single file (see below: Documentation about the most relevant result files), adds some variables, and applies some filters, and computes summaries of the results (such as mean error, MSE, coverage, etc.).
- All following files starting with 4-, 5-, 6-, etc. generate the result plots, tables, and additional analyses
- dataFiles stores intermediate/preprocessed data files. For more details about the most relevant data files, see below "Documentation about the most relevant result files")
- helpers contains some helper functions as well as some mostly unstructured supplementary analyses. These are not well documented.
- Appendices contains the two appendices
- MA-methods contains all meta-analytical methods. All return their results in the same structured way. Please note that they have been implemented for two-group t-tests, and might or might not work for other types of tests.
- sim-studies contains the code for simulating the raw meta-analytical data with different true underlying effects, levels of QRP, levels of publication bias, etc. The single most important function is called
simMA, which generates
kprimary studies with certain settings of QRPs, publication bias, true effect size, heterogeneity, etc.
- oldStuff The file oldStuff/hilgard_result_table.R is helpful for breaking down the results into more readable parcels. Other old analyses are stored in this folder.
- Plots contains the plots of the publication
Documentation about the most relevant result files
Unaggregated data files (in /dataFiles)
- res.final.RData: This file contains the most fine-grained results of all meta-analytic (MA) techniques in long format; each row is one result of one MA technique for one simulated data set. No filters have been applied.
- res.wide.RData: This file contains the most fine-grained results of all meta-analytic (MA) techniques in wide format; each row is one MA technique for one simulated data set. No filters have been applied.
- res.wide.red.RData: A copy of res.wide (in a previous version, we applied some filters to res.wide, but this is not necessary any more). All further summaries have been computed based on this data set.
- hyp.wide.RData: This is generate as a summary file for the hypothesis test plot (see 5-hypTestPlot.R) and also used in the Shiny app.
These files are too large for Github. You can generate them yourself by running 3-resultsFramework.R. Be aware that doing so requires a lot of RAM (it works with 16GB)!
Aggregated data files (in /dataFiles)
- summ.RData / summ.csv: This is the summary file which contains ME, RMSE, coverage, etc. for each method and each condition. Most result plots and tables are based on this file.
Extending the analyses, reusing the simulated meta-analyses
We took a lot of care to simulate the meta-analyses in a way that they are plausible concerning QRPs, publication bias, samples sizes, etc. When new meta-analytical methods are developed, we encourage researchers to use these simulated files and conditions as sort of benchmark data to test their methods. This allows an easy comparison of the results.
conditions.txt lists all 432 conditions with their parameter settings. That means, you can pick single conditions of interest.
0.2 (2018/02/01): tagged as "revision submitted to journal"
> sessionInfo() R version 3.4.1 (2017-06-30) Platform: x86_64-apple-darwin15.6.0 (64-bit) Running under: macOS High Sierra 10.13.3 Matrix products: default BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib locale:  de_DE.UTF-8/de_DE.UTF-8/de_DE.UTF-8/C/de_DE.UTF-8/de_DE.UTF-8 attached base packages:  parallel stats4 compiler stats graphics grDevices utils datasets methods base other attached packages:  bindrcpp_0.2 weightr_1.1.2 puniform_0.0.2 broom_0.4.2 notifyR_1.02 rjson_0.2.15 RCurl_1.95-4.8 bitops_1.0-6  progress_1.1.2 metafor_2.0-0 Matrix_1.2-11 meta_4.8-4 doMC_1.3.4 iterators_1.0.8 foreach_1.4.3 data.table_1.10.4-3  dplyr_0.7.4 reshape2_1.4.2 truncdist_1.0-2 evd_2.3-2 truncnorm_1.0-7 pwr_1.2-1 MASS_7.3-47 loaded via a namespace (and not attached):  Rcpp_0.12.15 pillar_1.1.0 plyr_1.8.4 bindr_0.1 prettyunits_1.0.2 tools_3.4.1 tibble_1.4.2 nlme_3.1-131 lattice_0.20-35  pkgconfig_2.0.1 rlang_0.1.6 psych_1.7.8 stringr_1.2.0 grid_3.4.1 glue_1.2.0 R6_2.2.2 foreign_0.8-69 purrr_0.2.4  tidyr_0.7.2 magrittr_1.5 codetools_0.2-15 mnormt_1.5-5 assertthat_0.2.0 stringi_1.1.5
0.1 (2017/05/20): tagged as "submitted to journal & preprint"
> sessionInfo() R version 3.1.3 (2015-03-09) Platform: x86_64-suse-linux-gnu (64-bit) Running under: SUSE Linux Enterprise Server 11 SP4 locale:  LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8  LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C  LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages:  parallel stats4 compiler stats graphics grDevices utils datasets methods base other attached packages:  puniform_0.0.2 rio_0.5.0 gtools_3.5.0 doParallel_1.0.10 broom_0.4.2 notifyR_1.02 rjson_0.2.15 RCurl_1.95-4.8  bitops_1.0-6 progress_1.1.2 metafor_1.9-6 Matrix_1.2-7.1 meta_4.8-1 doMC_1.3.4 iterators_1.0.8 foreach_1.4.3  data.table_1.10.4 dplyr_0.5.0 reshape2_1.4.1 truncdist_1.0-2 evd_2.3-2 truncnorm_1.0-7 pwr_1.2-1 MASS_7.3-47 loaded via a namespace (and not attached):  assertthat_0.2.0 chron_2.3-50 codetools_0.2-15 curl_2.6 DBI_0.6-1 devtools_1.13.0 digest_0.6.9 foreign_0.8-67  git2r_0.18.0 grid_3.1.3 haven_1.0.0 httr_1.2.1 jsonlite_1.4 lattice_0.20-34 magrittr_1.5 memoise_1.1.0  mnormt_1.5-5 nlme_3.1-128 openxlsx_3.0.0 plyr_1.8.4 prettyunits_1.0.2 psych_1.7.5 R6_2.2.1 Rcpp_0.12.10  readODS_1.6.4 readxl_1.0.0 rstudioapi_0.6 stringi_1.0-1 stringr_1.2.0 tibble_1.3.0 tidyr_0.6.2 tools_3.1.3  withr_1.0.2 XML_3.98-1.7