Bootstrap test for two correlated curves

[frederiquefrouin]

When testing Delong versus bootstrap using alternative "less", I obtained quite different results for p-values. It seems to be an error in bootstrap (?) (1-p-value ?)

roctest<-roc.test(roc1, roc2, method="d", alternative="less")
print(roctest)

DeLong's test for two correlated ROC curves

data: roc1 and roc2
Z = -1.9582, p-value = 0.0251
alternative hypothesis: true difference in AUC is less than 0
sample estimates:
AUC of roc1 AUC of roc2
0.7388633 0.5391705

roctest<-roc.test(roc1, roc2, method="b", alternative="less")
|====================================================================================| 100%
print(roctest)

Bootstrap test for two correlated ROC curves

data: roc1 and roc2
D = 1.971, boot.n = 2000, boot.stratified = 1, p-value = 0.9756
alternative hypothesis: true difference in AUC is less than 0
sample estimates:
AUC of roc1 AUC of roc2
0.7388633 0.5391705

---

[xrobin]

Thanks for your report. Can you please provide the data you used to generate these ROC curves as requested in the bug report template?

[frederiquefrouin]

Data_1.xlsx
Data_2.xlsx

roc1<-roc(Data_1$class, Data_1$V1)
roc2<-roc(Data_2$Class, Data_2$V2)
roctest<-roc.test(roc1, roc2, method="d", alternative="less")
print(roctest)
roctest<-roc.test(roc1, roc2, method="b", alternative="less")
print(roctest)

[frederiquefrouin]

Data associated with the issue

[xrobin]

This is very useful, thank you. Here is a code snippet to fully reproduce:

library(xlsx)
Data_1 <- read.xlsx("Data_1.xlsx", 1)
Data_2 <- read.xlsx("Data_2.xlsx", 1)
roc1<-roc(Data_1$Class, Data_1$V1)
roc2<-roc(Data_2$Class, Data_2$Val1)
roctest<-roc.test(roc1, roc2, method="d", alternative="less")
print(roctest)
roctest<-roc.test(roc1, roc2, method="b", alternative="less")
print(roctest)

Although there is no guarantee that the two tests should return the same value, this seems definitely more different that I would expect, and I can't see an obvious reason why the values would be reversed. See:

roc.test(roc1, roc2, method="b", alternative = "greater")

I will investigate this in more detail as soon as I can.

[xrobin]

I finally had time to look at the issue. It turns out the reason is that the curves have direction=">". This is clearly not handled properly by the delong test in pROC and needs to be fixed.

[xrobin]

This issue is now fixed in the development version of pROC, and the delong test returns the correct result for these ROC curves:

> roctest<-roc.test(roc1, roc2, method="d", alternative="less")
> print(roctest)

	DeLong's test for two correlated ROC curves

data:  roc1 and roc2
Z = 1.9582, p-value = 0.9749
alternative hypothesis: true difference in AUC is less than 0
sample estimates:
AUC of roc1 AUC of roc2 
  0.7388633   0.5391705 

Please let me know if you think this doesn't solve your problem.

Thanks again for your report and for helping me make pROC better!