FAQ Frequently asked questions

The X axis should be "1-Specificity" instead of "Specificity"!

This is not a bug!

Most statistical tools cannot plot specificity from 1 to 0, so instead they plot 1-specificity from 0 to 1. This results in exactly the same plot, just with a different labelling of the axis.

R doesn't have this limitation, and pROC plots specificity from 1 to 0, saving you from doing a few subtractions when you look at the plot. If you are using pROC in S+, you only have the choice to plot 1-specificity.

If you are concerned that your reviewers will be confused and really wish to have the "old-style" 1-Specificity from 0 to 1, or if you plan to change the axis label to "False Positive Rate" (which is really 1-specificity), you can use the legacy.axis argument to the plot function. See ?plot.roc for more details.

I have a GLM/SVM/PLS/Whatever model, how can I analyze it with pROC?

You need to use the predict() function to get predictions that can be passed to pROC (as the predictor argument).

Can I create a ROC curve with two predictors? Can I adjust fixed / random effects in pROC?

Not directly, but you can! You need to use a modeling function (such as glm()) or package (such as lme4); fit and check the model, then use the predict() function to get predictions that can be passed to pROC (as the predictor argument).

My dataset is huge. Can I use pROC?

A large effort has been made to render pROC more efficient with large ROC curves. An algorithm with complexity of O(n) has been rolled out with pROC 1.6 and can be used with algorithm=2 argument to roc

system.time(roc(round(runif(1E6)), rnorm(1E6), algorithm=2))

This only takes about 20 seconds on my laptop machine. The DeLong test has also been made more efficient and the memory footprint was removed.

If your dataset is much larger, it might help to use the old algorithm that goes in O(n**t*), where t is the number of thresholds that can be measured in your data. You can reduce the number of thresholds by rounding your predictor so that if you keep about a 1000 unique predictor values you still get a good approximation of your ROC curve. Use algorithm=3 for a faster C++ implementation with Rcpp.

system.time(roc(round(runif(1E6)), round(rnorm(1E6), digits=1), algorithm=3))

You can play around with algorithm=0 that runs a microbenchmark and chooses the fastest algorithm for your data.

Can I compute the confidence interval of the best threshold of the curve (not its SE/SP)?

Yes you can since pROC 1.6 and its ci.coords function. For instance:

> ci.coords(aSAH$outcome, aSAH$s100b, x="best", input = "threshold", ret="threshold")
95% CI (2000 stratified bootstrap replicates):
           2.5%   50%  97.5%
threshold 0.115 0.205 0.4854

There is 95% chance that the optimal threshold lies between 0.115 and 0.4854.

The functions are giving me weird error messages I don't understand

Several packages on CRAN provide alternative roc or auc functions. These packages can interfere with pROC, especially if they are loaded later in the session (and hence appear earlier in the search path).

For instance, here are a few messages you may see if you have the AUC package loaded:

Not enough distinct predictions to compute area under the ROC curve.

Error in roc(outcome ~ ndka, data = aSAH) : unused argument (data = aSAH)

If that happens, you should unload the package by typing detach("package:AUC"). Alternatively you can refer to the pROC version of the function specifically through their namespace:

pROC::auc(pROC::roc(aSAH$outcome, aSAH$ndka))

Here is a list of packages providing roc and auc functions, possibly conflicting with pROC:

library(sos)
auc.search = findFn("auc") 
auc.search[auc.search$Function == "auc", c("Package", "Function", "Description", "Link")]

Package	Function	Description and Link
PresenceAbsence	auc	Area Under the Curve
AUC	auc	Compute the area under the curve of a given performance...
spatstat	auc	Area Under ROC Curve
PK	auc	Estimation of confidence intervals for the area under the...
aucm	auc	AUC
SDMTools	auc	Area Under the Curve of the Reciever Operating Curve
StatMeasures	auc	Area under curve of predicted binary response
precrec	auc	Retrieve a data frame of AUC scores
flux	auc	Calculate the area under a line(curve).
grpreg	auc	Calculates AUC for cv.grpsurv objects
ncvreg	auc	Calculates AUC for cv.ncvsurv objects
enrichvs	auc	Function to calculate the Area Under the ROC Curve (AUC)
FFTrees	auc	Calculates AUC (Area under the Curve) using trapezoidal...
radiant.model	auc	Area Under the Curve (AUC)
Metrics	auc	Compute the area under the ROC (AUC)
MXM	auc	ROC and area under the curve
MESS	auc	Compute the area under the curve for two vectors.
kulife	auc	Compute the area under the curve for two vectors.
asbio	auc	Area under a reciever operating characterisitic (ROC) curve
ModelMetrics	auc	Area Under the Curve

roc.search = findFn("roc") 
roc.search[roc.search$Function == "roc", c("Package", "Function", "Description", "Link")]

Package	Function	Description and Link
analogue	roc	ROC curve analysis
MAMSE	roc	Receiver Operating Characteristic (ROC) Curves
spatstat	roc	Receiver Operating Characteristic
AUC	roc	Compute the receiver operating characteristic (ROC) curve.
sdm	roc	plot ROC curves
NEArender	roc	ROC for NEA benchmarks
fmsb	roc	Calculate Receiver Operating Characteristic (ROC) curve
epiDisplay	roc	ROC curve
aucm	roc	ROC and AUC