ggplot.resamples and plot.resamples when results constant

[c1au6i0]

Using the function plot or ggplot for an object of class resamples will not visualize data points for a model in which data$model$values are constant or near constant.
The problem seems to be in these lines of code of the ggplot.resamples function (line 982):

results <- lapply(
      plotData,
      function(x, cl) {
        ttest <- try(t.test(x$value, conf.level = cl),
                     silent = TRUE)
        if (class(ttest)[1] == "htest") {
          out <- c(ttest$conf.int, ttest$estimate)
          names(out) <-
            c("LowerLimit", "UpperLimit", "Estimate")
        } else
          out <- rep(NA, 3)
        out
      }

In particular the t.test function will fail to report confidence intervals or means. An example:

t.test(rep(1,10), conf.level = 0.95)

Error in t.test.default(rep(1, 10), conf.level = 0.095) : data are essentially constant 

and thus also the mean alone will not be plotted and the out will be vector of NA.

I suppose that a model with results nearly constant is a rare occurrence, but I realized this because it happened to me. :)

Claudio

[topepo]

So maybe return

out <- c(LowerLimit = NA_real_, UpperLimit = NA_real_, Estimate = mean(x$value, na.rm = TRUE))

instead?

[c1au6i0]

Tested,
that works.

Thanks :)

[topepo]

Do you have a test case that I can use for a unit test?

[c1au6i0]

uhm,
would something like this work?

library(mlbench)
data(Sonar)
set.seed(998)

# one of the variables is the same as the classifier
Sonar[,"Vx"] <- Sonar$Class

fitControl <- trainControl(## 10-fold CV
                           method = "repeatedcv",
                           number = 5,
                           repeats = 5)

gbmFit1 <- train(Class ~ ., data = Sonar, 
                 method = "gbm", 
                 trControl = fitControl,
                 verbose = FALSE)


resamps <- resamples(list(gbmFit1, gbmFit1))

ggplot(resamps)

[topepo]

Give those changes a try. It works for the example that I cooked up.