DeLong AUC Confidence Interval

[anaphylacticapps]

I think there may be an issue with the DeLong confidence interval for AUC. When the sample size gets large, the CI goes to either 0-0 or 1-1. Here is an example:

Create ROC Objects

predictor1 <- c(runif(12000,0,0.5), runif(14472-12000, 0.5,0.75))
response1 <- rbinom(14472, size=1, p=predictor1)
roc1 <- roc(response1, predictor1)

predictor2 <- c(runif(3 * 12000,0,0.5), runif(3 * (14472-12000), 0.5,0.75))
response2 <- rbinom(3 * 14472, size=1, p=predictor2)
roc2 <- roc(response2, predictor2)

predictor3 <- c(runif(10 * 12000,0,0.5), runif(10 * (14472-12000), 0.5,0.75))
response3 <- rbinom(10 * 14472, size=1, p=predictor3)
roc3 <- roc(response3, predictor3)

Calculate AUC and CI

auc(roc1)
Area under the curve: 0.7586
ci.auc(roc1)
95% CI: 0.7506-0.7667 (DeLong)

auc(roc2)
Area under the curve: 0.7584
ci.auc(roc2)
95% CI: 0.7537-0.7631 (DeLong)

auc(roc3)
Area under the curve: 0.7561
ci.auc(roc3)
95% CI: 1-1 (DeLong)

[xrobin]

@sieste could you please have a look at this? It looks like the new DeLong code is returning erroneous theta values, maybe an overflow or something like that?

> pROC:::delongPlacementsCpp(roc3)$theta
[1] 15.97072

@anaphylacticapps as workaround, the old version 1.8 of pROC reports correct results, although it takes significantly more time to do so:

> ci.auc(roc3)
95% CI: 0.7567-0.7618 (DeLong)
> pROC:::delongPlacementsCpp(roc3)$theta
[1] 0.7592556

[sieste]

That should do the trick, sorry about that. I didn't expect someone would need a confidence interval at a sample size so big that it causes integer overflow ;)

Cheers, Stefan

[xrobin]

Excellent, thanks a lot for the very quick fix!

This is the same as commit e71e57b then. I'll add a check on the R side to make sure theta == auc to catch similar cases in the future. Then push a bugfix release asap.

[anaphylacticapps]

Thank you, both.

[xrobin]

There is now an error message when the DeLong AUC doesn't match.

Thanks @anaphylacticapps for the report!