automatic learning curves

[mheilman]

It would be nice to have functionality to automatically run experiments with different size training sets in order to plot performance as a function of training sample size.

Perhaps this could be a separate experiment type (e.g., like "evaluate"). This probably does not make sense for cross-validation.

Perhaps there should be an option not to save all the models, etc., since there could be hundreds.

Possible configuration options:

• a list of sample sizes to consider (the default could be powers of 2 starting at 32 or 64)
• the number of replications per sample size (this could default to 1, but higher values would produce smoother learning curves)

(This was also briefly mentioned in the discussion of #212.)

[desilinguist]

This should use the learning curves feature from scikit-learn: http://scikit-learn.org/stable/modules/generated/sklearn.learning_curve.learning_curve.html

[desilinguist]

@langep, has there been any progress on this issue?

[desilinguist]

I'd like to include this in the upcoming v1.3.

Here's how I am thinking about this:

1. Have a new experiment type called learning_curves which will compute the learning curve using scikit-learn's built-in method and save the output as a CSV in the results directory. It will also save the actual learning curve plot as a PNG in the results directory if matplotlib is available.
2. The models for the various training sizes will not be saved.
3. Users will be able to specify the various training sizes and the number of cross-validation iterations to be used for averaging.
4. Since we generally want to do at least 10 folds of cross-validation to get a smooth learning curve, grid search will not be allowed within each fold since that would make it too slow.

Since this will be a new feature, I'd like to solicit input from all of you: @dan-blanchard @dmnapolitano @mheilman @bndgyawali @mulhod @benbuleong @aoifecahill @aloukina @cml54 @bwriordan.

Thanks!

[desilinguist]

addressed by #332.