Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bootstrap test for two correlated curves #64

Closed
frederiquefrouin opened this issue Oct 29, 2019 · 6 comments
Closed

Bootstrap test for two correlated curves #64

frederiquefrouin opened this issue Oct 29, 2019 · 6 comments
Labels
bug

Comments

@frederiquefrouin
Copy link

@frederiquefrouin frederiquefrouin commented Oct 29, 2019

When testing Delong versus bootstrap using alternative "less", I obtained quite different results for p-values. It seems to be an error in bootstrap (?) (1-p-value ?)

roctest<-roc.test(roc1, roc2, method="d", alternative="less")
print(roctest)

DeLong's test for two correlated ROC curves

data: roc1 and roc2
Z = -1.9582, p-value = 0.0251
alternative hypothesis: true difference in AUC is less than 0
sample estimates:
AUC of roc1 AUC of roc2
0.7388633 0.5391705

roctest<-roc.test(roc1, roc2, method="b", alternative="less")
|====================================================================================| 100%
print(roctest)

Bootstrap test for two correlated ROC curves

data: roc1 and roc2
D = 1.971, boot.n = 2000, boot.stratified = 1, p-value = 0.9756
alternative hypothesis: true difference in AUC is less than 0
sample estimates:
AUC of roc1 AUC of roc2
0.7388633 0.5391705


@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Oct 29, 2019

Thanks for your report. Can you please provide the data you used to generate these ROC curves as requested in the bug report template?

@frederiquefrouin

This comment has been minimized.

Copy link
Author

@frederiquefrouin frederiquefrouin commented Nov 1, 2019

Data_1.xlsx
Data_2.xlsx

roc1<-roc(Data_1$class, Data_1$V1)
roc2<-roc(Data_2$Class, Data_2$V2)
roctest<-roc.test(roc1, roc2, method="d", alternative="less")
print(roctest)
roctest<-roc.test(roc1, roc2, method="b", alternative="less")
print(roctest)

@frederiquefrouin

This comment has been minimized.

Copy link
Author

@frederiquefrouin frederiquefrouin commented Nov 1, 2019

Data associated with the issue

@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Nov 1, 2019

This is very useful, thank you. Here is a code snippet to fully reproduce:

library(xlsx)
Data_1 <- read.xlsx("Data_1.xlsx", 1)
Data_2 <- read.xlsx("Data_2.xlsx", 1)
roc1<-roc(Data_1$Class, Data_1$V1)
roc2<-roc(Data_2$Class, Data_2$Val1)
roctest<-roc.test(roc1, roc2, method="d", alternative="less")
print(roctest)
roctest<-roc.test(roc1, roc2, method="b", alternative="less")
print(roctest)

Although there is no guarantee that the two tests should return the same value, this seems definitely more different that I would expect, and I can't see an obvious reason why the values would be reversed. See:

roc.test(roc1, roc2, method="b", alternative = "greater")

I will investigate this in more detail as soon as I can.

@xrobin xrobin added bug and removed more-info-needed labels Nov 1, 2019
@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Nov 21, 2019

I finally had time to look at the issue. It turns out the reason is that the curves have direction=">". This is clearly not handled properly by the delong test in pROC and needs to be fixed.

xrobin added a commit that referenced this issue Nov 22, 2019
xrobin added a commit that referenced this issue Nov 22, 2019
@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Nov 22, 2019

This issue is now fixed in the development version of pROC, and the delong test returns the correct result for these ROC curves:

> roctest<-roc.test(roc1, roc2, method="d", alternative="less")
> print(roctest)

	DeLong's test for two correlated ROC curves

data:  roc1 and roc2
Z = 1.9582, p-value = 0.9749
alternative hypothesis: true difference in AUC is less than 0
sample estimates:
AUC of roc1 AUC of roc2 
  0.7388633   0.5391705 

Please let me know if you think this doesn't solve your problem.

Thanks again for your report and for helping me make pROC better!

@xrobin xrobin closed this Nov 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.