Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci.coords returns a "ret" error for input = "threshold" #57

Closed
pgcudahy opened this issue Jun 18, 2019 · 4 comments

Comments

@pgcudahy
Copy link

commented Jun 18, 2019

Describe the bug
When trying to use ci.coords with input="threshold" I get an error regarding ret="threshold"

To Reproduce

values <- c(81.3, 123.3, 42.9, 49.3, 31.9, 12.6, 23.3, 2, 165.8, 3.1, 13.6, 13.6, 10.4, 3.7, 2, 23.8, 47.5, 126.8, 10.3, 18.4)
categories <- c(1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1)
df <- data.frame(values=values, categories=categories, stringsAsFactors = FALSE)
roc1 <- roc(df$categories, df$values, data=df, plot = FALSE)
coords(roc1, 5, input="threshold")
ci.coords(roc1, 5, input="threshold")

Gives the error

Error in ci.coords.roc(roc1, 5, input = "threshold"): 'threshold' output is only supported for best ROC point ('x = "best"')
Traceback:

1. ci.coords(roc1, 5, input = "threshold")
2. ci.coords.roc(roc1, 5, input = "threshold")
3. stop("'threshold' output is only supported for best ROC point ('x = \"best\"')")

Expected behavior
A confidence interval for sensitivity and specificity at the chosen threshold.

Additional context
I can see why choosing a return threshold generates an error. But this is an input threshold.

@xrobin

This comment has been minimized.

Copy link
Owner

commented Jun 18, 2019

This is because you have an implied "threshold" in the default value of ret.

This will work right away, setting the return value without threshold:

> ci.coords(roc1, 5, input="threshold", ret=c("se", "sp"))
  |======================================================================| 100%
95% CI (2000 stratified bootstrap replicates):
                         2.5%    50% 97.5%
threshold 5: sensitivity  0.8 0.9333     1
threshold 5: specificity  0.2 0.6000     1 

The threshold itself won't vary.

However in this case it should indeed be possible to return a (constant) threshold. I will investigate why it isn't the case.

In addition the default value should probably be able to auto-detect whether 'threshold' can be computed.

@xrobin xrobin added the bug label Jun 18, 2019

@pgcudahy

This comment has been minimized.

Copy link
Author

commented Jun 18, 2019

Thanks! Your solution works for me. I'm still a little confused, since a threshold is given by the user, what does it mean for the function to return a threshold, other than just show the input value? Shouldn't ret=c("se", "sp") be the default if the input is a threshold?

@xrobin

This comment has been minimized.

Copy link
Owner

commented Jun 18, 2019

I'm not sure if it should be the default. It's useful to ping it back in the case where many thresholds are given as input.

xrobin added a commit that referenced this issue Jun 18, 2019

xrobin added a commit that referenced this issue Jun 18, 2019

xrobin added a commit that referenced this issue Jun 18, 2019

@xrobin

This comment has been minimized.

Copy link
Owner

commented Jun 18, 2019

Fixed. Threshold is kept by default whenever possible, and removed from the return values otherwise.
I will attempt to push this to CRAN as 1.15.1 soon.

@xrobin xrobin closed this Jun 18, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.