Exception has been thrown by the target of an invocation. #2716

Symbai · 2023-06-15T11:03:24Z

System Information (please complete the following information):

Model Builder Version (available in Manage Extensions dialog):
Visual Studio Version 17.7.0 Preview 2.0

Describe the bug

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 154   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()

To Reproduce

I created a console project using .NET 7
I added a ML scenario for "Text Classification"
I added my CSV data (only 10 rows)
I selected the columns
I clicked on "Train"
A popup appears showing me the training time etc. looks all good
Exception occurs

Expected behavior
I expect to proceed to "Evaluate" step. If I manually go to this tab and press the "Predict" button, I get the same exception as above.

Screenshots

Additional context
Last log entry in VS is Generate code behind files

The text was updated successfully, but these errors were encountered:

LittleLittleCloud · 2023-06-15T17:57:43Z

Are you training on GPU, and would you mind sharing your mbconfig

Symbai · 2023-06-16T03:33:13Z

I was testing both

mbconfig:

{
  "Scenario": "TextClassification",
  "DataSource": {
    "Type": "TabularFile",
    "Version": 1,
    "FilePath": "C:\\Users\\User\\Desktop\\test.csv",
    "Delimiter": ",",
    "DecimalMarker": ".",
    "HasHeader": false,
    "ColumnProperties": [
      {
        "ColumnName": "col0",
        "ColumnPurpose": "Label",
        "ColumnDataFormat": "String",
        "IsCategorical": false,
        "Type": "Column",
        "Version": 4
      },
      {
        "ColumnName": "col1",
        "ColumnPurpose": "Text",
        "ColumnDataFormat": "String",
        "IsCategorical": false,
        "Type": "Column",
        "Version": 4
      }
    ]
  },
  "Environment": {
    "Type": "LocalCPU",
    "Version": 1
  },
  "RunHistory": {
    "Version": 3,
    "Type": "Result",
    "Trials": [
      {
        "Version": 1,
        "Type": "Trial",
        "TrainerName": "TextClassificationMulti",
        "Score": 0.1,
        "RuntimeInSeconds": 5.194,
        "Parameter": {
          "_SCHEMA_": "e0 * e1 * e2",
          "e0": {
            "OutputColumnName": "col0",
            "InputColumnName": "col0",
            "AddKeyValueAnnotationsAsText": false
          },
          "e1": {
            "LabelColumnName": "col0",
            "Sentence1ColumnName": "col1",
            "ScoreColumnName": "Score",
            "OutputColumnName": "PredictedLabel",
            "BatchSize": 32,
            "MaxEpochs": 10,
            "Architecture": "Roberta"
          },
          "e2": {
            "OutputColumnName": "PredictedLabel",
            "InputColumnName": "PredictedLabel"
          }
        }
      }
    ],
    "Estimators": {
      "e0": "MapValueToKey",
      "e1": "TextClassificationMulti",
      "e2": "MapKeyToValue"
    },
    "Schema": "e0 * e1 * e2",
    "MetricName": "MacroAccuracy",
    "ModelFilePath": "C:\\Users\\User\\source\\repos\\MlTest\\MlTest\\MLModel2.mlnet"
  },
  "Type": "TrainingConfig",
  "Version": 4,
  "TrainingOption": {
    "Version": 1,
    "Type": "TextClassficationTrainingOption",
    "TrainingTime": 2147483647,
    "LabelColumn": "col0",
    "TextColumn": "col1",
    "ValidationOption": {
      "Version": 0,
      "Type": "CrossValidationValidationOption",
      "NumberOfFolds": 5
    }
  }
}

bandicam.2023-06-16.05-29-19-286.mp4

LittleLittleCloud · 2023-06-16T06:24:31Z

Yea please. Thanks!

daikoz · 2023-07-10T20:36:07Z

same issue with same scenario with:
ML.NET version 17.17.0.2332602
Microsoft Visual Studio 2022 Version 17.6.4

Symbai · 2023-09-27T17:32:57Z

I am just experiencing it again on a different project. All I did was choosing a new TSV file and clicked on 'Train again'. But I also created a brand new project and got the same immediately on first train. Here is some updated info:

Exception:

   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.ML.Runtime.ComponentCatalog.LoadableClassInfo.CreateInstanceCore(Object[] ctorArgs)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes](IHostEnvironment env, Type signatureType, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.Runtime.ComponentCatalog.TryCreateInstance[TRes,TSig](IHostEnvironment env, TRes& result, String name, String options, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModelCore[TRes,TSig](IHostEnvironment env, TRes& result, Object[] extra)
   at Microsoft.ML.ModelLoadContext.TryLoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, Entry ent, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModelOrNull[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelLoadContext.LoadModel[TRes,TSig](IHostEnvironment env, TRes& result, RepositoryReader rep, String dir, Object[] extra)
   at Microsoft.ML.ModelOperationsCatalog.Load(Stream stream, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelOperationsCatalog.Load(String filePath, DataViewSchema& inputSchema)
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.SetTorchRunTimeFolderAndLoadModel(ITrainingConfiguration configuration, String modelPath, MLContext& context, ITransformer& model, DataViewSchema& inputSchema) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 139
   at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.GenerateConsumptionAsync(ITrainingConfiguration configuration, String trainingConfigurationFolder, String nameSpace, String className, TargetType target, String[] labels, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 155   at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.ML.ModelBuilder.ViewModels.TrainViewModel.<GenerateCodeBehindFilesAsync>d__100.MoveNext()

VS Log:

Set log file path to C:\Users\XXX\AppData\Local\Temp\MLVSTools\logs\MLModel1-YC9ZBU.txt
start text classification 
restore "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" --configfile "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\NuGet.config" -r win-x64 /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj"
publish "C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj" -r win-x64 -c Release --no-self-contained -o "C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5" --no-restore /p:UsingToolXliff=false /p:TorchSharpVersion=0.99.5 /p:TorchSharpCudaRuntimeVersion=1.13.0.1 /p:TensorflowRuntimeVersion=2.3.1 /p:BaseOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\\" /p:BaseIntermediateOutputPath="C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\obj\\"
start installing runtime in C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5
  Determining projects to restore...
  Restored C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj (in 418 ms).
MSBuild version 17.8.0-preview-23418-03+0125fc9fb for .NET
C:\Program Files\dotnet\sdk\8.0.100-rc.1.23455.8\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(311,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\PROGRAM FILES\MICROSOFT VISUAL STUDIO\2022\PREVIEW\COMMON7\IDE\EXTENSIONS\AEOWIWMR.5OO\AUTOMLSERVICE\RuntimeManager\torchsharp.gpu.csproj]
  torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\bin\Release\netstandard2.0\win-x64\torchsharp.gpu.dll
  torchsharp.gpu -> C:\Users\XXX\AppData\Local\Temp\ModelBuilder\torchsharp-cuda-0.99.5\
install runtime successfully
Use cross validation with fold: 5
|      Trainer                             MacroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     TextClassificationMulti             0,0102     463,1580       |
|--------------------------------------------------------------------|
|                          Experiment Results                        |
|--------------------------------------------------------------------|
|                               Summary                              |
|--------------------------------------------------------------------|
|ML Task: text classification                                        |
|Dataset: H:\xxxx.tsv                                        |
|Label : col0                                                        |
|Total experiment time :   463,1580 Secs                             |
|Total number of models explored: 1                                  |
|--------------------------------------------------------------------|
|                        Top 1 models explored                       |
|--------------------------------------------------------------------|
|      Trainer                             MacroAccuracy Duration    |
|--------------------------------------------------------------------|
|0     TextClassificationMulti             0,0102     463,1580       |
|--------------------------------------------------------------------|
Generate code behind files

Log file:

(Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update TrainViewModel_163.TrainingDidNotSucceed (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TryOutRow (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.TopFiveRecommendationLabel (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutItemIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update EvaluateViewModel_196.IsTryOutUserIdValueInDataset (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0788 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowTable (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:17.0973 DEBUG update ForecastingTryOutViewModel_197.IsShowGraph (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG fail calling update handler GetSampleCodeSnippetAsync (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG Exception has been thrown by the target of an invocation. (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
2023-09-27 19:25:21.2015 DEBUG    at StreamJsonRpc.JsonRpc.<InvokeCoreAsync>d__151`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ML.ModelBuilder.ToolWindows.ConsumeViewModel.<GetSampleCodeSnippetAsync>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<>c__DisplayClass15_0.<<Initialize>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.ML.ModelBuilder.ViewModel.CodeGenerator.DependencyGraphManager.<TryExecuteNotifyDependencyNodeDelegateAsync>d__20.MoveNext() (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)

LittleLittleCloud · 2023-09-27T17:46:02Z

Looks like you are running with cross-validation? The fix for cross-validation hasn't been fix yet so you might need to use another validation strategy

Symbai · 2023-09-28T04:58:11Z

Unbelievable, you're right. I switched from cross-validation (which seems to be the default option because I never changed that) to split-validation and this fixed it. But cross-validation seems to be the better one, any ETA when it will be fixed? Or can you at least say if the team is working on it?

//edit: With the first run using split-validation the results were acceptable. Now I've trained again and I get completely different prediction results comparing to the first run. Not just slightly different values but completely(!) different results. Even though the data which is used for training hasn't changed.

LittleLittleCloud · 2023-09-28T05:09:04Z

@Symbai Actually, looks like the issue has already been fixed by me,,,, can you try upgrade model builder to v17.17.* and give it another trial?

Symbai · 2023-09-28T05:53:00Z

The VS extension I have says its the latest version: 17.17.0.2332602 see screenshot in my previous post #2716 (comment)

//edit: Reinstalled the extension but still the same

LittleLittleCloud · 2023-09-28T08:13:47Z

OK, let me double check then

Symbai · 2023-10-06T15:52:42Z

@LittleLittleCloud any update?

LittleLittleCloud · 2023-10-06T17:35:46Z

@Symbai No haven't yet, I was focusing on other stuff the last week. Will get the update back to this thread once I have any.

Also feel free to ping me for reminder

hannespreishuber · 2023-12-19T16:16:45Z

same here
Use cross validation with fold: 5

Trainer MicroAccuracy Duration
0 TextClassificationMulti 0,3045 65,9930

LittleLittleCloud self-assigned this Jun 15, 2023

LittleLittleCloud added the Bug Something isn't working label Jun 15, 2023

Symbai mentioned this issue Jun 16, 2023

Text-Classification: Invocation exception when cross-validation #2718

Closed

github-actions bot added the Stale label Aug 10, 2023

github-actions bot removed the Stale label Sep 29, 2023

github-actions bot added the Stale label Nov 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception has been thrown by the target of an invocation. #2716

Exception has been thrown by the target of an invocation. #2716

Symbai commented Jun 15, 2023

LittleLittleCloud commented Jun 15, 2023 •

edited

Symbai commented Jun 16, 2023 •

edited

LittleLittleCloud commented Jun 16, 2023

daikoz commented Jul 10, 2023

Symbai commented Sep 27, 2023

LittleLittleCloud commented Sep 27, 2023

Symbai commented Sep 28, 2023 •

edited

LittleLittleCloud commented Sep 28, 2023 •

edited

Symbai commented Sep 28, 2023 •

edited

LittleLittleCloud commented Sep 28, 2023

Symbai commented Oct 6, 2023

LittleLittleCloud commented Oct 6, 2023

hannespreishuber commented Dec 19, 2023

Exception has been thrown by the target of an invocation. #2716

Exception has been thrown by the target of an invocation. #2716

Comments

Symbai commented Jun 15, 2023

LittleLittleCloud commented Jun 15, 2023 • edited

Symbai commented Jun 16, 2023 • edited

mbconfig:

LittleLittleCloud commented Jun 16, 2023

daikoz commented Jul 10, 2023

Symbai commented Sep 27, 2023

LittleLittleCloud commented Sep 27, 2023

Symbai commented Sep 28, 2023 • edited

LittleLittleCloud commented Sep 28, 2023 • edited

Symbai commented Sep 28, 2023 • edited

LittleLittleCloud commented Sep 28, 2023

Symbai commented Oct 6, 2023

LittleLittleCloud commented Oct 6, 2023

hannespreishuber commented Dec 19, 2023

LittleLittleCloud commented Jun 15, 2023 •

edited

Symbai commented Jun 16, 2023 •

edited

Symbai commented Sep 28, 2023 •

edited

LittleLittleCloud commented Sep 28, 2023 •

edited

Symbai commented Sep 28, 2023 •

edited