AutoML Regression Experiment Crash

[superichmann]

System Information (please complete the following information):

• OS & Version: Windows 10
• ML.NET Version: ML.NET v2.0.1 automl 0.20.1
• .NET Version: e.g. .NET 6.0

Describe the bug
AutoML Experiment crash

To Reproduce
I am doing autoML on data from database.. but attached is the csv of that data
automlbug.csv
split train test fraction -> 0.01
100 seconds experiment
target column sales
preFeaturizer->doubletosingle
Optimization metric: RegressionMetric.MeanAbsoluteError

Expected behavior
experiment should not crash

Screenshots, Code, Sample Projects

    System.AggregateException: One or more errors occurred. (Index was outside the bounds of the array.) ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.

  Stack Trace: 
    PipelineProposer.ProposeSearchSpace()
    EciCostFrugalTuner.Propose(TrialSettings settings)
    AutoMLExperiment.RunAsync(CancellationToken ct)
    --- End of inner exception stack trace ---
    Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
    Task`1.GetResultCore(Boolean waitCompletionNotification)
    AutoMLExperiment.Run()
    RegressionExperiment.Execute(IDataView trainData, IDataView validationData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)

[michaelgsharp]

@LittleLittleCloud can you take a look at this.

[LittleLittleCloud]

@superichmann Can you try the latest nightly build? We have a few fix regarding to this "index out of bounds) in nightly build

[superichmann]

yes but I have other issue with nightly :[ #6654

[LittleLittleCloud]

KK, I'll close this one then as the bug related to this issue has been fixed already

[superichmann]

I still get this error on latest nightly :|

System.AggregateException: One or more errors occurred. (Index was outside the bounds of the array.)
 ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Microsoft.ML.AutoML.PipelineProposer.ProposeSearchSpace()
   at Microsoft.ML.AutoML.EciCostFrugalTuner.Propose(TrialSettings settings)
   at Microsoft.ML.AutoML.AutoMLExperiment.RunAsync(CancellationToken ct)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.ML.AutoML.AutoMLExperiment.Run()
   at Microsoft.ML.AutoML.RegressionExperiment.Execute(IDataView trainData, IDataView validationData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
   at Microsoft.ML.AutoML.RegressionExperiment.Execute(IDataView trainData, IDataView validationData, String labelColumnName, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)

[LittleLittleCloud]

Interesting,, I try the automlbug.csv in model builder which is basically a wizard for AutoML and everything just work. Could you share the code somewhere so I can reproduce the bug

[superichmann]

bug_test_2.csv
bug_train_2.csv
Originally from database... if it matters.

var experimentSettings = new RegressionExperimentSettings();
experimentSettings.MaxModels = 30;
experimentSettings.OptimizingMetric = DEFAULT_REGRESSION_METRIC;
experimentSettings.CacheBeforeTrainer = CacheBeforeTrainer.Off;
experimentSettings.CacheDirectoryName = null;
RegressionExperiment experiment = mlContext.Auto().CreateRegressionExperiment(experimentSettings);
// here create preFeaturizer that converts each Double column to Single
experimentResult = experiment.Execute(bug_train_2, bug_test_2, "sales", preFeaturizer: preDoubleToSingle); // ERROR HERE

[LittleLittleCloud]

@superichmann
Maybe it's because of preDoubleToSingle? I run the following code and everything just work.

            var context = new MLContext(1);
            var trainPath = @"D:\bug_train_2.csv";
            var testPath = @"D:\bug_test_2.csv";
            var label = "sales";
            var columnInference = context.Auto().InferColumns(trainPath, label);
            var textLoader = context.Data.CreateTextLoader(columnInference.TextLoaderOptions);
            var trainData = textLoader.Load(trainPath);
            var testData = textLoader.Load(testPath);
            var settings = new RegressionExperimentSettings
            {
                MaxModels = 30,
                OptimizingMetric = RegressionMetric.MeanAbsoluteError,
                CacheBeforeTrainer = CacheBeforeTrainer.Off,
            };

            var result = context.Auto()
                .CreateRegressionExperiment(settings)
                .Execute(trainData, testData, label);

            Assert.True(result.BestRun.ValidationMetrics.MeanAbsoluteError > 0);

[superichmann]

mmm I don't know anymore :| I have to use the preFeaturizer remember? from #6595 I will try to change some other settings hopefully it will fix it :\

[superichmann]

see newer comment for precise reproduction
Still no fix :|
I have managed to reproduce an error which might happen due to the original exception here..

Please try these reproduction steps:

1. Download these datasets:
   tr.csv
   te.csv
2. update mlnet cli to latest alpha version - in my windows machine the path C:\Users\uzer\.dotnet\tools\.store\mlnet-win-x64\16.15.1\mlnet-win-x64\16.15.1\tools\net6.0\any contains Microsoft.ML.dll version 3.0.23.20401 and Microsoft.ML.AutoML.dll version 0.2100.23.20401
3. run mlnet regression --dataset tr.csv --validation-dataset te.csv --label-col sales --cache Off --log-file-path mlogError.txt
4. this should produce models with NaN score (all of them) and eventually show error System.TimeoutException: Training time finished without completing a successful trial. Either no trial completed or the metric for all completed trials are NaN or Infinity

Maybe this can help you hone in on the bug @LittleLittleCloud if not tell me and I will try to provide additional info.