Understanding the different curves and scores

It is not clear that some metrics like pLI or CCR have a bimodal distribution with most benign variants lying at the extreme low ends of their distributions and pathogenic variants at the higher end. In other metrics like CADD, a sharp decline in scoring capability can be seen just outside of the cutoff window which can be seen from the J curves.

The J curve is a different interpretation of a ROC curve, it is true positive rate (TPR) minus false positive rate (FPR) vs the normalized metric score from 0 to 1, and ROCs are simply TPR vs FPR. ROC curves may show best how metrics score all variants but their AUCs (area under curve) are colored by the set of true negatives provided.

A PR curve is precision, or positive predictive value (), vs recall, which is TPR. PR curves can deal with the issue of true negative influence by focusing primarily on a metric’s ability to correctly identify true positives. Clinical Utility tells a user summarily both how accurate a tool is at scoring variants in a gene and how many it is capable of scoring at all. The majority of variant pathogenicity predictor papers neglect to include PR curves, J curves or Clinical Utility scores, but these are often used by diagnosticians when determining the appropriate metric to use for evaluating their variant sets. Each tool can provide a different kind of insight into a metric’s performance on a particular truth set or subset, and whether it is the right metric to use for a specific cohort.