Invalid seeds for boot632 train control method

[asardaes]

Simple example:

data(iris)
TrainData <- iris[,1:4]
TrainClasses <- iris[,5]

knnFit1 <- train(TrainData, TrainClasses,
                 method = "knn",
                 preProcess = c("center", "scale"),
                 tuneLength = 10,
                 trControl = trainControl(method = "boot632"))

Results in:

Error in { : task 2 failed - "supplied seed is not a valid integer"

---

Cause:

Within the nominalTrainWorkflow function the nested foreach loops have arguments iter = seq(along = resampleIndex) and parm = 1:nrow(info$loop). However, when ctrl$method is "boot632" the resampleIndex variable changes and gets the AllData argument appended at the beginning. This causes the foreach loops to do one extra iteration.

The last element of the seeds list is a single numeric value, but if length(parm) > 1 then the following code:

if (!(length(ctrl$seeds) == 1 && is.na(ctrl$seeds))) 
            set.seed(ctrl$seeds[[iter]][parm])

... attempts to access a non-existing value, therefore doing set.seed(NA), which gives the error mentioned above.

---

Solution:

I'm not sure about this but I think that the seeds list must also get M values in the last element when ctrl$method == "boot632", where M is the number of models as specified in the documentation.

[topepo]

You hit the nail on the head. It does an extra model fit on the entire data set for that method. I'll fix that and update this thread.

Thanks,

Max

[asardaes]

I checked this and it seems to work if the seeds parameter is not provided. Since I was working around the problem, I noticed now this doesn't work:

data(iris)
TrainData <- iris[,1:4]
TrainClasses <- iris[,5]

num_resamples <- 5
num_models <- 5

# specify seeds
set.seed(329)
seeds <- sample(1000, num_resamples*num_models + 1)
seeds <- lapply(seq(from = 1, by = num_models, to = length(seeds)), 
                function(i_seed) seeds[i_seed:(i_seed+num_models)])
seeds[[length(seeds)]] <- seeds[[length(seeds)]][1]

Working

# this one expects a list of length 'num_resamples + 1' (call it L)
knnFit1 <- train(TrainData, TrainClasses,
                 method = "knn",
                 preProcess = c("center", "scale"),
                 tuneLength = num_models,
                 trControl = trainControl(method = "boot", 
                                          number = num_resamples, 
                                          seeds = seeds))

Not working

# this one fails, it expects a list of length L+1
knnFit2 <- train(TrainData, TrainClasses,
                 method = "knn",
                 preProcess = c("center", "scale"),
                 tuneLength = num_models,
                 trControl = trainControl(method = "boot632", 
                                          number = num_resamples, 
                                          seeds = seeds))

---

If the second case should indeed receive a longer seeds list, then I guess the documentation should reflect it.