evaluateModelFit broken?

[dvrbts]

I downloaded 3.0-4 earlier this week, and now a function that used to work no longer does.

shosh.hmsc.rand.eval <- evaluateModelFit(hM=shosh.hmsc.rand,
+ predY=shosh.hmsc.rand.preds)
Error in roc.default(response, predictor, auc = TRUE, ...) : 
  No control observation.

This command worked fine recently. If I foolishly repeat the command after the first warning R crashes and I lose eveything from that session.

[oysteiop]

Hi, this function has not changed recently. Could you give us some details about your model structure?

[jarioksa]

Well, the function was changed on Oct 24 (commit cfb442c), and it seems that this changed line may be involved in this error, but I cannot spot any obvious reason for the error in this line. However, the error is triggered in imported code from other packages: Hmsc has no direct call to roc, but this is probably in pROC code imported in evaluateModelFit. However, the last CRAN update of pROC was in July, so that the package has not changed recently.

I cannot reproduce the error. Can you give detailed traceback() after the error or step through the function (with debugonce(evaluateModelFit)) so that we find the ultimate HMSC line that gives the error?

Is the shosh.hmsc.rand saved from a previous session? The old result objects may not be compatible with the new code, and perhaps you should re-calculate the result instead of re-using old results.

[dvrbts]

Jari et al., It's definitely in roc. Here's a reproducble result from a clean workspace. Dave data(shoshveg) shoshveg <- dropspc(shoshveg,4) shoshveg <- shoshveg > 0 shoshveg <- as.matrix(shoshveg) data(shoshsite) shoshsite <- shoshsite[,c(2,4:10,23)] shosh.hmsc <- Hmsc(Y=shoshveg,XData=shoshsite,distr='probit') samples <- 1000 transient <- 1000 thin <- 5 nChains <- 2 verbose <- 500 shosh.hmsc <- sampleMcmc(shosh.hmsc,samples=samples, + transient=transient, thin=thin, + nChains=nChains, verbose=verbose) shosh.hmsc.preds <- computePredictedValues(shosh.hmsc) debugonce(evaluateModelFit(hM=shosh.hmsc, + predY=shosh.hmsc.preds)) Error in roc.default(response, predictor, auc = TRUE, ...) : No control observation. traceback() 7: stop("No control observation.") 6: roc.default(response, predictor, auc = TRUE, ...) 5: auc.default(Y[sel, i], predY[sel, i], levels = c(0, 1), direction="<") 4: auc(Y[sel, i], predY[sel, i], levels = c(0, 1), direction = "<") 3: computeAUC(hM$Y[, sel, drop = FALSE], mPredY[, sel, drop = FALSE]) 2: evaluateModelFit(hM = shosh.hmsc, predY = shosh.hmsc.preds) 1: debugonce(evaluateModelFit(hM = shosh.hmsc, predY = shosh.hmsc.preds))

…

On 11/25/19 8:39 AM, Jari Oksanen wrote: Well, the function was changed on Oct 24, and it seems that this changed line may be involved in this error, but I cannot spot any obvious reason for the error in this line. However, the error is triggered in imported code from other packages: *Hmsc* has no direct call to |roc|, but this is probably in *pROC* code imported in |evaluateModelFit|. However, the last CRAN update of *pROC* was in July, so that has not changed recently. I cannot reproduce the error. Can you give detailed |traceback()| after the error or step through the function (with |debugonce(evaluateModelFit)|) so that we find the ultimate *HMSC* line that gives the error? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=ADZXMMXYCV2JU6J6N2HIBNTQVN6LJA5CNFSM4JRC4ZOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFBN5CY#issuecomment-558030475>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZXMMX4RKAIRQ4BO4NNSV3QVN6LJANCNFSM4JRC4ZOA>.

-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ David W. Roberts Professor Department of Ecology, Montana State University On sabbatical at: Swiss Federal Research Institute WSL, Zuercherstrasse 111 CH-8903 Birmensdorf, Switzerland

[jarioksa]

Dave,

The error really seems to be triggered by this Oct 24 commit (cfb442c) which explicitly sets levels = c(0,1) for pROC::auc(): These are not the levels in your data, but you have c(FALSE, TRUE). In general, these are interchangeable in mathematical operations, but the pROC package in one place identifies the control cases as as.character(levels[1]) and "0" and "FALSE" are different names. So it is a peculiarity in the pROC package that was exposed by the change in Hmsc::evaluateModelFit().

I think it is prudent to see how to handle this smoothly within Hmsc. However, an immediate solution is to take care that your data are numeric 0 or 1. Instead of

shoshveg <- shoshveg > 0

do

shoshveg <- ifelse(shoshveg > 0, 1, 0)

or any alternative that turns the logical condition from FALSE and TRUE to numeric 0 and 1. This solved the problem with me when I ran your example.

[gtikhonov]

Hi Dave and Jari,

As it was me, who made this commit, I believe that I should provide some explainantion. The previous version of this function was returning result=max(auc,1-auc) due to the somewhat non-obvious behaviour of the used auc / roc function. Apparently, unless you explicitly specify it, the order of your binary values is chosen based on the predicted scores - the one with higher mean predictive score is defined as "positive" and the other one as "negative". I fixed this misbehaviour and, therefore, in my opinion, the new version is not really bugged with respect to this aspect.

I completely agree that with binary data, a boolean type is natural. However, I do not have an clear view of how to generalise the call to auc function, so it would properly handle both 0/1 and FALSE/TRUE options. Any thoughts?

[dvrbts]

Jari, Thanks; that's an elegant solution. Doing as.numeric(shoshveg) converts to 0/1 but also to a vector, as opposed to a matrix, while ifelse maintains the structure. Thanks, Dave

…

On 11/25/19 1:25 PM, Jari Oksanen wrote: Dave, The error really seems to be triggered by this Oct 24 commit (cfb442c <cfb442c>) which explicitly sets |levels = c(0,1)|k for |pROC::auc()|: These are not the levels in your data, but you have |c(FALSE, TRUE)|. In general, these are interchangeable in mathematical operations, but the *pROC* package in one place identifies the cases as |as.character(levels[1])| and |"0"| and |"FALSE"| are different names. So it is a peculiarity in the *pROC* package that was exposed by the change in |Hmsc::evaluateModelFit()|. I think it is prudent to see how to handle this smoothly within *Hmsc*. However, an immediate solution is to take care that your data are numeric 0 or 1. Instead of shoshveg <- shoshveg > 0 do shoshveg <- ifelse(shoshveg > 0,1,0) or any alternative that turns the logical condition from |FALSE| and |TRUE| to numeric |0| and |1|. This solved the problem with me when I ran your example. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=ADZXMMXDHTFEEOO3D4POK3DQVO72PA5CNFSM4JRC4ZOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFCG32I#issuecomment-558132713>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZXMMVQ6YJLY5SOPEUT2CTQVO72PANCNFSM4JRC4ZOA>.

-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ David W. Roberts Professor Department of Ecology, Montana State University On sabbatical at: Swiss Federal Research Institute WSL, Zuercherstrasse 111 CH-8903 Birmensdorf, Switzerland

[dvrbts]

Gleb, Since it preserves numeric 0/1 results as well as converts FALSE/TRUE, I think wrapping the argument inside evaluateModelFit to ifelse(Y>0,1,0) seems low cost and effective. Dave

…

On 11/25/19 1:48 PM, Gleb Tikhonov wrote: Hi Dave and Jari, As it was me, who made this commit, I believe that I should provide some explainantion. The previous version of this function was returning |result=max(auc,1-auc)| due to the somewhat non-obvious behaviour of the used |auc| / |roc| function. Apparently, unless you explicitly specify it, the order of your binary values is chosen based on the predicted scores - the one with higher mean predictive score is defined as "positive" and the other one as "negative". I fixed this misbehaviour and, therefore, in my opinion, the new version is not really bugged with respect to this aspect. I completely agree that with binary data, a boolean type is natural. However, I do not have an clear view of how to generalise the call to |auc| function, so it would properly handle both |0/1| and |FALSE/TRUE| options. Any thoughts? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=ADZXMMVYBUOUATH7T4L2DI3QVPCRDA5CNFSM4JRC4ZOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFCIYWA#issuecomment-558140504>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZXMMW6O7L6OEUCHGYGKEDQVPCRDANCNFSM4JRC4ZOA>.

-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ David W. Roberts Professor Department of Ecology, Montana State University On sabbatical at: Swiss Federal Research Institute WSL, Zuercherstrasse 111 CH-8903 Birmensdorf, Switzerland

[jarioksa]

I implemented a solution that forces data into {0,1} for AUC in evaluateModelFit (commit 904ad54). An alternative solution would have been to force data into {FALSE,TRUE} and change the AUC call to pROC::auc(..., levels=c(FALSE,TRUE)). In my test this fixed the issue.