Auto train pipeline Bug

[taha-ghadirian]

System information

• OS version/distro: Windows server 2012r2 - windows 10
• .NET Version: .net core 3.1
• Ml.net Version: Last Stable Version.

Issue

• What did you do?: Set a auto Training pipeline using API, set a long MaxExperimentTimeInSeconds (For example 600) for small dataset.
• What happened?: Program hanged without any response.
• What did you expect?: the program should stop training after 600 seconds. and show me the result.
• Am I Tested it with shorter time?: Yes! I Tested it with 120Seconds and 100 Seconds. Its work currectly!

Source code / logs

var experimentSettings = new MulticlassExperimentSettings()
            {
                MaxExperimentTimeInSeconds = 600,
                CacheDirectory = new DirectoryInfo("cache"),
                CacheBeforeTrainer = CacheBeforeTrainer.Auto,
                OptimizingMetric = MulticlassClassificationMetric.MicroAccuracy,
            };

        var experiment = mlContext.Auto().CreateMulticlassClassificationExperiment(experimentSettings);

        var crossValidationExperimentResult = experiment.Execute(
            trainData: mappedInputData,
            labelColumnName: "Label",
            progressHandler: new Progress<RunDetail<MulticlassClassificationMetrics>>()
        );

[justinormont]

How many models were produced? The AutoML code returns between models; consequently a slow model causes the end time to be approximate.

Possible fix: (inside of AutoML)
Have AutoML use ML․NET's ctx.CancelExecution() function to interrupt the current model when the timer expires.

Perhaps something like:

Timer timer = new System.Timers.Timer(_experimentSettings.MaxExperimentTimeInSeconds * 1000);
timer.Elapsed += timerHandler;
timer.AutoReset = false;
timer.Enabled = true;

private void timerHandler(object sender, EventArgs e) {
    ctx.CancelExecution();
}

Placed before AutoML's do/while sweeping loop.

We may want to consider adding a parameter of MinIterations defaulting to 1, so we ensure that at least one model is created. Currently once AutoML starts the first model, the time expiring won't cancel it.

Refs:

• image classification needs cancel #4632
• Train operation should be cancellable. #844
• Add support for cancelling of training #2997