Add Trial Validation Split In AutoML Loop

[alexbrillant]

What it is

My pull request does: Add a trial split to the trial data container.

How it works

I coded it this way:

class Trial:
    """
    Trial data container for :class:`AutoML`.
    A Trial contains the results for each validation split.
    Each trial split contains both the training set results, and the validation set results.
    .. seealso::
        :class:`AutoML`,
        :class:`TrialSplit`,
        :class:`HyperparamsRepository`,
        :class:`BaseHyperparameterSelectionStrategy`,
        :class:`RandomSearchHyperparameterSelectionStrategy`,
        :class:`DataContainer`
    """

    def __init__(
            self,
            hyperparams: HyperparameterSamples,
            status: 'TRIAL_STATUS' = None,
            pipeline: BaseStep = None,
            validation_splits=None,
            cache_folder=None,
            error=None,
            error_traceback=None,
            start_time=None,
            end_time=None
    ):
        if status is None:
            status = TRIAL_STATUS.PLANNED
        if validation_splits is None:
            validation_splits = []

        self.status: TRIAL_STATUS = status
        self.hyperparams = hyperparams
        self.pipeline: BaseStep = pipeline
        self.validation_splits = validation_splits
        self.cache_folder = cache_folder
        self.error_traceback = error_traceback
        self.error = error
        self.start_time = start_time
        self.end_time = end_time

Example usage

Here is how it is used in the AutoML loop class:

 for trial_number in range(self.n_trial):
            self.print_func('\ntrial {}/{}'.format(trial_number + 1, self.n_trial))
            auto_ml_data = self._load_auto_ml_data(trial_number)

            with self.hyperparams_repository.new_trial(auto_ml_data) as repo_trial:

                for training_data_container, validation_data_container in validation_splits:
                    p = copy.deepcopy(self.pipeline)
                    p.update_hyperparams(repo_trial.hyperparams)
                    repo_trial.set_hyperparams(p.get_hyperparams())

                    with repo_trial.new_validation_split(p) as repo_trial_split:
                        try:
                            repo_trial_split = self.trainer.fit(
                                trial=repo_trial_split,
                                train_data_container=training_data_container,
                                validation_data_container=validation_data_container,
                                context=context
                            )

                            repo_trial_split.set_success()

                            self.print_func('trial {}/{} split {}/{} score: {}'.format(
                                trial_number + 1,
                                self.n_trial,
                                repo_trial_split.split_number + 1,
                                len(training_data_container),
                                repo_trial_split.get_validation_score()
                            ))

                        except Exception as error:
                            track = traceback.format_exc()
                            self.print_func(track)
                            repo_trial_split.set_failed(error)

                    repo_trial.update_final_trial_status()

            self.hyperparams_repository.save_trial(repo_trial)

        best_hyperparams = self.hyperparams_repository.get_best_hyperparams(

[guillaume-chevalier]

I've already made the comment in the bast, but it'd be good to provide some example comment for the time printed so that readers can realize how faster it is. Optional fix.

[guillaume-chevalier]

Maybe def __call__ instead? This would allow for passing functions as well as splitters. Idk.

[alexbrillant]

I have tried to do this, and failed. I think it's fine like this tbh