CV fold for ensembler after ensembling_indices split

[angela97lin]

Currently in AutoML, if we want to train an ensemble, we create an ensemble split (

evalml/evalml/automl/engine/engine_base.py

Line 220 in 16c30b8

X_train, y_train = X.iloc[automl_config.ensembling_indices], y.iloc[automl_config.ensembling_indices]

). This is to prevent overfitting, by not training the ensemble on the same data that the metalearners are trained on.

Then, in train_and_score_pipeline, we split the ensembling indices data using our data splitter and train/validate on one fold of the data (

evalml/evalml/automl/engine/engine_base.py

Line 145 in 16c30b8

if pipeline.model_family == ModelFamily.ENSEMBLE and i > 0:

). Is this necessary? The ensemble internally already does cross-validation. For small datasets, this means we're scoring on 1/3 * 0.2 (ensemble indices size) of data. For the happiness dataset with 128 rows (#2093), the mean_cv_score is calculated using just 8 rows. 😬 Perhaps we can remove these lines of code and just train the ensemble on the full ensemble_indices.

@dsherry @bchen1116 @rpeck FYI

[dsherry]

Marking this as blocked on #2093. Its possible the fix for #2093 includes fixing the code described here!

[angela97lin]

Closed by #2260