Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci.coords returns a "ret" error for input = "threshold" #57

Closed
pgcudahy opened this issue Jun 18, 2019 · 4 comments
Closed

ci.coords returns a "ret" error for input = "threshold" #57

pgcudahy opened this issue Jun 18, 2019 · 4 comments
Labels
bug No, it's not a feature!

Comments

@pgcudahy
Copy link

Describe the bug
When trying to use ci.coords with input="threshold" I get an error regarding ret="threshold"

To Reproduce

values <- c(81.3, 123.3, 42.9, 49.3, 31.9, 12.6, 23.3, 2, 165.8, 3.1, 13.6, 13.6, 10.4, 3.7, 2, 23.8, 47.5, 126.8, 10.3, 18.4)
categories <- c(1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1)
df <- data.frame(values=values, categories=categories, stringsAsFactors = FALSE)
roc1 <- roc(df$categories, df$values, data=df, plot = FALSE)
coords(roc1, 5, input="threshold")
ci.coords(roc1, 5, input="threshold")

Gives the error

Error in ci.coords.roc(roc1, 5, input = "threshold"): 'threshold' output is only supported for best ROC point ('x = "best"')
Traceback:

1. ci.coords(roc1, 5, input = "threshold")
2. ci.coords.roc(roc1, 5, input = "threshold")
3. stop("'threshold' output is only supported for best ROC point ('x = \"best\"')")

Expected behavior
A confidence interval for sensitivity and specificity at the chosen threshold.

Additional context
I can see why choosing a return threshold generates an error. But this is an input threshold.

@xrobin
Copy link
Owner

xrobin commented Jun 18, 2019

This is because you have an implied "threshold" in the default value of ret.

This will work right away, setting the return value without threshold:

> ci.coords(roc1, 5, input="threshold", ret=c("se", "sp"))
  |======================================================================| 100%
95% CI (2000 stratified bootstrap replicates):
                         2.5%    50% 97.5%
threshold 5: sensitivity  0.8 0.9333     1
threshold 5: specificity  0.2 0.6000     1 

The threshold itself won't vary.

However in this case it should indeed be possible to return a (constant) threshold. I will investigate why it isn't the case.

In addition the default value should probably be able to auto-detect whether 'threshold' can be computed.

@xrobin xrobin added the bug No, it's not a feature! label Jun 18, 2019
@pgcudahy
Copy link
Author

Thanks! Your solution works for me. I'm still a little confused, since a threshold is given by the user, what does it mean for the function to return a threshold, other than just show the input value? Shouldn't ret=c("se", "sp") be the default if the input is a threshold?

@xrobin
Copy link
Owner

xrobin commented Jun 18, 2019

I'm not sure if it should be the default. It's useful to ping it back in the case where many thresholds are given as input.

@xrobin
Copy link
Owner

xrobin commented Jun 18, 2019

Fixed. Threshold is kept by default whenever possible, and removed from the return values otherwise.
I will attempt to push this to CRAN as 1.15.1 soon.

@xrobin xrobin closed this as completed Jun 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug No, it's not a feature!
Projects
None yet
Development

No branches or pull requests

2 participants