Implemented optimism bootstrap

[asardaes]

Some time ago I implemented the optimism bootstrap to test it. I also added a method boot_all to get all bootstrap estimates in one run. If you want to include these, you can merge this branch. I added the reference to the documentation of trainControl. There is some more info in this post too.

I also added the .onUnload function, which should be used when a package uses compiled code (see good practice here)

[coveralls]

Coverage decreased (-0.03%) to 16.405% when pulling 78cc0d7 on asardaes:boot into 8456f03 on topepo:master.

[topepo]

Looks great. Thanks

[topepo]

Uh actually, there is a bug. The last call generated an error of Error in { : task 1 failed - "undefined columns selected".

library(caret)

set.seed(24)
dat <- twoClassSim(1000)

set.seed(34521)
opt_acc <- train(Class ~ ., data = dat, 
                 method = "svmRadial",
                 preProc = c("center", "scale"),
                 tuneLength = 10,
                 trControl=trainControl(method = "optimism_boot"))

set.seed(34521)
opt_acc_probs <- train(Class ~ ., data = dat, 
                       method = "svmRadial",
                       preProc = c("center", "scale"),
                       tuneLength = 10,
                       trControl=trainControl(method = "optimism_boot", 
                                              classProbs = TRUE))
set.seed(34521)
opt_roc <- train(Class ~ ., data = dat, 
                 method = "svmRadial",
                 preProc = c("center", "scale"),
                 tuneLength = 10,
                 metric = "ROC",
                 trControl=trainControl(method = "optimism_boot", 
                                        classProbs = TRUE,
                                        summaryFunction = twoClassSummary))

[asardaes]

In twoClassSummary there is the following line:

rocAUC <- ModelMetrics::auc(ifelse(data$obs == lev[2], 0, 1), data[, lvls[1]])

It seems that lvls[1] = "Class1", but the columns of data are "pred", "obs", "rowIndex" and "y". That's why there's a "undefined columns selected" error. I'm not sure if I should change that line, the data itself seems to be fine though.

EDIT: nevermind, I'm not adding the probValues columns correctly, I'll look into it.