You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have some problem interpreting the following graphs that plot Sensitivity vs Normalized Rank of a perfect model.
library(precrec)
p <- rbinom(100, 1, 0.5) # same vector for predictions and observations
prc <- evalmod(scores = p, labels = p, mode="basic")
autoplot(prc, c("Specificity", "Sensitivity"))
I would expect that a perfect model would generate values of Specificity = Sensitivity = 1 for all the retrieved ranked documents and thus, a line with slope 0 and intercept 1. I am clearly missing something and/or misinterpreting the x axis label. Any hint?
Thanks
The text was updated successfully, but these errors were encountered:
Yes, both specificity and sensitivity should be 1 for all the ranks as you mentioned. It seems like the calculation of normalised ranks is incorrect in this case.
The current version of precrec does not provide a clear and intuitive calculation method for ties when mode="basic" is specified. For example, rbinom(100, 1, 0.5) produces many tied scores that are either 0 or 1. The following 3 plots were created by precrec for a dataset generated by rbinom(100, 1, 0.5).
The red dots in the specificity and the sensitivity plots correspond to the red line in the ROC plot. Also, the red dots indicate that specificity stays at 1 while sensitivity changes from 0 to 1. This approach is not incorrect, and it is handy to calculate interpolated ROC and precision-recall curves. Nonetheless, the following plots can be more intuitive when there are tied scores and mode="basic" is used.
The plots above were generated in a way that the same ties have the same sensitivity and specificity. I need to look into whether or not this approach makes sense and how to implement it if so.
I have some problem interpreting the following graphs that plot Sensitivity vs Normalized Rank of a perfect model.
I would expect that a perfect model would generate values of Specificity = Sensitivity = 1 for all the retrieved ranked documents and thus, a line with slope 0 and intercept 1. I am clearly missing something and/or misinterpreting the x axis label. Any hint?
Thanks
The text was updated successfully, but these errors were encountered: