Systematic meta-review of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement
DOI: https://doi.org/10.1016/j.jclinepi.2023.04.012
Objectives: To 1) explore trends of risk of bias (ROB) in prediction research over time following key methodological publications, using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and 2) assess the inter-rater agreement of the PROBAST.
Study design and setting: PubMed and Web of Science were searched for reviews with extractable PROBAST scores on domain and signaling question (SQ) level. ROB trends were visually correlated with yearly citations of key-publications. Inter-rater agreement was assessed using Cohen’s Kappa.
Results: 139 systematic reviews were included, of which 85 reviews (containing 2477 single studies) on domain level and 54 reviews (containing 2458 single studies) on SQ level. High ROB was prevalent, especially in the Analysis domain, and overall trends of ROB remained relatively stable over time. The inter-rater agreement was low, both on domain (Kappa 0.04–0.26) and SQ level (Kappa -0.14–0.49).
Conclusion: Prediction model studies are at high ROB and time trends in ROB as assessed with the PROBAST remain relatively stable. These results might be explained by key-publications having no influence on ROB or recency of key-publications. Moreover, the trend may suffer from the low inter-rater agreement and ceiling effect of the PROBAST. The inter-rater agreement could potentially be improved by altering the PROBAST or providing training on how to apply the PROBAST.
- 1.data_prep-20221104.R: Loading in and preparing data
- 2.data_analysis-20221104.Rmd: Performing analyses