Include more explicit comparison of replication success depending on criterion

[LukasWallrich]

Patrick Smela shared that success criterion drastically changes replication success rate - we should report that very clearly (in both annotator and explorer) so that users can report in a meaningful way. @LukasRoeseler

[LukasWallrich]

Replication Bayes Factor
Small telescopes
Prediction interval
2 x Significance

Frame as distance-based vs conclusion-based

[LukasWallrich]

Currently, result variable based on significance - remove and rename that var throughout?

[LukasRoeseler]

Listing our options and some thoughts for future discussion:

• 2 x Significance: currently the main way of presentation but problematic due to more and more original studies being n.s. and different alpha levels being used (maybe allow users to set alpha level)
• Signal/No Signal terminology from: listed below the main results and based on Lebel et al. (https://doi.org/10.15626/MP.2018.843)
• Replication Bayes Factor
• Small telescopes: I understand that this is a sample size planning heuristic (N_replication = 2.5 x N_original, maybe he meant something with d_33%?
• Prediction interval
• Relative effect size: depicted in ReD data paper, probably the easiest way if we do not want to categorize data, ignores precision of effects

[LukasWallrich]

• explain consistency as simply original in ci replication in popup

• significance: show replication sig when OS not significant (split grey segment in 2)

• Ask Mark re prediction intervals

• allow to set global alpha level?

On hold:

• Bayes Factors

Out:

• Small telescopes not as success criterion (could be power criterion)?

[LukasRoeseler]

Prediction Interval Computation (from the supplementary materials of Patil et al., 2016 and suggested by Patrick Smela):

roriginal +/- z0.975*sqrt( (1/noriginal-3) + (1/nreplication-3) ).

Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers
expect when they replicate studies? A statistical view of
replicability in psychological science. Perspectives on Psychological
Science, 11(4), 539–544. https://doi.org/10.1177/1745691616646366

[LukasWallrich]

That actually seems very sensible - see ChatGPT explanation here - but it rewards imprecise original effects, given that they are compatible with most subsequent results. At least I would struggle with accepting this as a success criterion for n.s. orig studies. So, def worth implementing as an option - but maybe with n.s. orig studies at least separated out?

[LukasWallrich]

Schafmeister (2022) has quite a good explanation of the criteria - particularly relevant as it explains how small telescopes can be a binary criterion (and I would be ok with introducing it in that way) - also contains refs to BF, and might generally be a good template for explaining the different criteria.

"An intuitive criterion relies on the binary decision of whether the replication produces evidence that is statistically significant at the 5% level in the same direction as the original finding. The main point of contention in using this criterion is its inability to quantify the strength of the evidence contained in a replication coded as failed or successful. For example, a failed replication attempt might not be able to detect a true effect if it is itself underpowered, whereas a high-powered replication might uncover an effect that is orders of magnitude smaller than the original effect. The “small-telescopes” approach of Simonsohn (2015) is based on the idea of detectability in that it tests whether the effect obtained in a replication study is smaller than an effect size that the original study had only 33% power to detect. In comparison, the Bayesian approach applied by Etz (2015) is based on the calculation of Bayes factors (BFs) comparing the hypothesis of no effect with an alternative of the effect size found in the original publication, originally proposed by Verhagen and Wagenmakers (2014)."

[LukasRoeseler]

Adding the recent discussion and proposal from Lakens here:

• Textbook description with code (17.3): https://lakens.github.io/statistical_inferences/17-replication.html#analyzing-replication-studies
• Bsky thread: https://bsky.app/profile/giladfeldman.bsky.social/post/3kvt64ecy632y

[LukasRoeseler]

Code for the heterogeneity test suggested by Andersen and Maxwell that works with FReD data:

# create function to compare effect sizes
rtest <- function(n1, n2, r1, r2) {
  r.1 <- metafor::escalc(ni = n1, ri = r1, measure = "COR")
  r.2 <- metafor::escalc(ni = n2, ri = r2, measure = "COR")
  
  metadata <- data.frame(yi = c(r.1$yi, r.2$yi), 
                         vi = c(r.1$vi, r.2$vi), 
                         study = c("original", "replication"))
  
  p <- rma(yi, 
            vi, 
            data = metadata, 
            method = "FE")
  
  
  return(c(p$QEp, p$I2))
  rm(list(r.1, r.2, metadata, p))
}

# compute variables for dataset
red$heterogeneity_pval <- rtest(r1 = red$es_original, r2 = red$es_replication, n1 = red$n_original, n2 = red$n_replication)[1]
red$heterogeneity_i2 <- rtest(r1 = red$es_original, r2 = red$es_replication, n1 = red$n_original, n2 = red$n_replication)[2]

[LukasWallrich]

Great! Maybe we can create a set of confusion matrices at some point to see how the different classifiers diverge? Maybe based on an extended function assess_replication_success() that returns results for all the different options in a tidy format?

[LukasRoeseler]

• make function work on multiple rows (works with for-loop but not with mapply)
• implement function into Annotator (also check if it harmonizes with plots)
• implement function into Explorer (also check if it harmonizes with plots)
• possibly revise function outputs (they are very specific now and do not simply say "success" or "failure")
• add more options

[LukasRoeseler]

Preliminary comparison (order is a bit confusing, but you can get a first grasp of what id successful and what not):

Edit: Now sorted. The different sorting and this plot should not be very important for the App btw as users will choose one outcome criterion and the Plotly output will let you hover over the categories and tell you what they mean. This is just so that you can estimate how strongly the criteria differ with respect to the average success rate.