Add Nested Cross-Validation

[topepo]

See http://www.biomedcentral.com/1471-2

[ck37]

Looks like this link no longer works, but maybe this was the article? http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3994246/
"Cross-validation pitfalls when selecting and assessing regression and classification models"

[enissay14]

http://stats.stackexchange.com/questions/189364/caret-repeated-k-fold-cross-validation-vs-nested-k-fold-cross-validation-repe

[jorjasso]

I think would be nice if some checks are added for nested cross-validation

• stability of the predictions across the best models in the outer loop
• stability of the hyperparameters
  Take a look in these links:
  http://jmlr.org/papers/v11/cawley10a.html

http://stats.stackexchange.com/questions/65128/nested-cross-validation-for-model-selection/65156#65156

[achekroud]

just curious, is the nested CV feature still on the cards? I think this would be awesome

[jesalo]

Nested CV would be a very useful - in particular for cases with small amounts of data. Is the feature clashing with the structure of caret?

[topepo]

Nested resampling would require a pretty substantial workflow that needs to be created and it is on the back-burner right now. However, rsample can generate the resampling indices. I'll try to take a simple example and make a vignette for that package.

[topepo]

I did a blog post instead.

[topepo]

This will not be implemented.