Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failing to pair roc curves builed with formula #68

Closed
Marc-Flo opened this issue Dec 20, 2019 · 4 comments
Closed

failing to pair roc curves builed with formula #68

Marc-Flo opened this issue Dec 20, 2019 · 4 comments
Labels
bug

Comments

@Marc-Flo
Copy link

@Marc-Flo Marc-Flo commented Dec 20, 2019

Hello !

I'm currently working on a database whicj contains missing values for the predictors.
When applying are.paired function to two roc objects built with the formula method, i get a negative result...
I have tried to apply the are.paired function to the same roc object built with the default method and this time i get a positive result.

Is there any reason that can explain why the roc object built with the formula method aren't recognized as paired ?

Thank you !

@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Dec 20, 2019

Works for me:

> data(aSAH)
> roc1 <- roc(outcome ~ wfns, aSAH)
Setting levels: control = Good, case = Poor
Setting direction: controls < cases
> roc2 <- roc(outcome ~ ndka, aSAH)
Setting levels: control = Good, case = Poor
Setting direction: controls < cases
> are.paired(roc1, roc2)
[1] TRUE

Please provide the steps to reproduce as requested when you created this bug report.

@Marc-Flo

This comment has been minimized.

Copy link
Author

@Marc-Flo Marc-Flo commented Dec 21, 2019

Here's an exemple of the code i used yesterday :

test <- data.frame(
  response = c(1,0,1,0,0,0,1,0,0,0,0,0,0,1,1,0,0,0,1,1),
  pred_1 = c(15,0,30,32,0,10,64,NA,10,0,0,0,0,0,108,0,0,0,56,22),
  pred_2 = c(0.7,0.1,0.2,0.2,1.4,0.1,0.2,1.9,0.8,0.4,NA,0.5,0.6,NA,3.7,1.7,0.3,0.7,0.7,NA)
  )

stand_1 <- roc(test$response, test$pred_1, algorithm = 1, ci = T)
stand_2 <- roc(test$response, test$pred_2, algorithm = 1, ci = T)

are.paired(stand_1, stand_2)

form_1 <- roc(response ~ pred_1, data = test, algorithm = 1, ci = T)
form_2 <- roc(response ~ pred_2, data = test, algorithm = 1, ci = T)

are.paired(form_1, form_2) 
@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Dec 22, 2019

Thanks for the update.
It's probably due to model.frame taking care of the NAs by itself. It works again if you set it to pass NAs, and let the main roc method take care of them.

na_1 <- roc(response ~ pred_1, data = test, algorithm = 1, ci = T, na.action=na.pass)
na_2 <- roc(response ~ pred_2, data = test, algorithm = 1, ci = T, na.action=na.pass)
are.paired(na_1, na_2) 
[1] TRUE

I will see how this can be handled. At the moment I'm thinking setting a default na.action=na.pass and documenting the behavior if a different na.action is used, but there might be a better way.

@xrobin xrobin added the bug label Dec 22, 2019
xrobin added a commit that referenced this issue Dec 23, 2019
… or fail (issue #68)
@xrobin

This comment has been minimized.

Copy link
Owner

@xrobin xrobin commented Dec 23, 2019

It's working now. The na.action is set to na.pass internally. Nothing changed from a user's perspective, except a warning if you pass a different na.action (typically na.omit or na.exclude).

> form_1 <- roc(response ~ pred_1, data = test, algorithm = 1, ci = T)
> form_2 <- roc(response ~ pred_2, data = test, algorithm = 1, ci = T)
> 
> are.paired(form_1, form_2) 
[1] TRUE

It will be available in pROC 1.16 that I'm going to release within the next couple of weeks. Thanks again for the report!

@xrobin xrobin closed this Dec 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.