Schema mismatch for input column 'Image': expected image, got Vector<Byte> for Predictions made using model trained with MobilenetV2 and ResnetV250

[adriansd27]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): .NET Core 3.1

Issue

I'm running an ASP.NET Core 3.1 Web app with ML.NET. I'm unable to make predictions with trained model when using ImageClassificationTrainer.Architecture.MobilenetV2 and ImageClassificationTrainer.Architecture.ResnetV250.

However it works with ImageClassificationTrainer.Architecture.InceptionV3 and ImageClassificationTrainer.Architecture.ResnetV2101.

• What did you do?
  I've trained the model. Then I've made a prediction with prediction engine pool.
• What happened?
  Got a mismatch error when trying to predict.
• What did you expect?
  I'd like to be able to make predictions with MobilenetV2 and ResnetV250 as well.

Source code / logs

            var trainDataRawBytes = _mapper.Map<IEnumerable<ImageDataRaw>>(imageData);
            var testDataRawBytes = _mapper.Map<IEnumerable<ImageDataRaw>>(testData);

            trainingData = _mlContext.Data.LoadFromEnumerable(trainDataRawBytes);

            var options = new ImageClassificationTrainer.Options
            {
                FeatureColumnName = "Image",
                LabelColumnName = "LabelKey",
                Arch = ImageClassificationTrainer.Architecture.MobilenetV2,
                Epoch = 50,       //100
                BatchSize = 10,
                LearningRate = 0.01f
            };

            IEstimator<ITransformer> pipeline =
                _mlContext.Transforms.Conversion.MapValueToKey("LabelKey", "Label")
                    .Append(_mlContext.MulticlassClassification.Trainers
                        .ImageClassification(options))
                    .Append(_mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabelValue",
                        "PredictedLabel"))
                    .AppendCacheCheckpoint(_mlContext);

            model = pipeline.Fit(trainingData);
            testingResults = EvaluateModel(testDataRawBytes);`

        private List<ImagePredictionResult> EvaluateModel(IEnumerable<ImageDataRaw> imageData)
        {
            return imageData.ToList().Select(image => _predictionEnginePool.Predict("ImageClassificationModel", image))
                .ToList().Select(
                    imagePrediction => new ImagePredictionResult
                    {
                        RealLabel = imagePrediction.Label,
                        PredictedLabel = imagePrediction.PredictedLabelValue,
                        Score = imagePrediction.Score.Max()
                    }).ToList();
        }

   at Microsoft.ML.Transforms.Image.ImageResizingTransformer.CheckInputColumn(DataViewSchema inputSchema, Int32 col, Int32 srcCol)
   at Microsoft.ML.Data.OneToOneTransformerBase.CheckInput(DataViewSchema inputSchema, Int32 col, Int32& srcCol)
   at Microsoft.ML.Data.OneToOneTransformerBase.OneToOneMapperBase..ctor(IHost host, OneToOneTransformerBase parent, DataViewSchema inputSchema)
   at Microsoft.ML.Transforms.Image.ImageResizingTransformer.Mapper..ctor(ImageResizingTransformer parent, DataViewSchema inputSchema)
   at Microsoft.ML.Transforms.Image.ImageResizingTransformer.MakeRowMapper(DataViewSchema schema)
   at Microsoft.ML.Data.RowToRowTransformerBase.Microsoft.ML.ITransformer.GetRowToRowMapper(DataViewSchema inputSchema)
   at Microsoft.ML.Data.TransformerChain`1.Microsoft.ML.ITransformer.GetRowToRowMapper(DataViewSchema inputSchema)
   at Microsoft.ML.PredictionEngineBase`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at Microsoft.ML.PredictionEngine`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at Microsoft.ML.ModelOperationsCatalog.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at Microsoft.Extensions.ML.PredictionEnginePoolPolicy`2.Create()
   at Microsoft.Extensions.ObjectPool.DefaultObjectPool`1.Create()
   at Microsoft.Extensions.ObjectPool.DefaultObjectPool`1.Get()
   at Microsoft.Extensions.ML.PredictionEnginePool`2.GetPredictionEngine(String modelName)
   at Microsoft.Extensions.ML.PredictionEnginePoolExtensions.Predict[TData,TPrediction](PredictionEnginePool`2 predictionEnginePool, String modelName, TData example)
   at PDCSBE.Services.Implementation.TrainService.<EvaluateModel>b__9_0(ImageDataRaw image) in D:\...\TrainService.cs:line 175
   at System.Linq.Enumerable.SelectListIterator`2.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at PDCSBE.Services.Implementation.TrainService.EvaluateModel(IEnumerable`1 imageData) in D:\...\TrainService.cs:line 175
   at PDCSBE.Services.Implementation.TrainService.TrainModel(IEnumerable`1 imageData, IEnumerable`1 testData, Int32 modelId) in D:\...\TrainService.cs:line 106
   at PDCSBE.Services.Implementation.TrainService.TrainModelWithMetrics(Int32 modelId) in D:\...\TrainService.cs:line 50
   at PDCSBE.Api.Controllers.TrainController.TrainModel(Int32 modelId) in D:\...\TrainController.cs:line 24
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Logged|12_1(ControllerActionInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)```

Please paste or attach the code or logs or traces that would be helpful to diagnose the issue you are reporting.

[kere-nel]

Hi @adriansd27,

I'm not able to reproduce. Can you provide the whole project?

[mstfbl]

Hi @adriansd27 , can you please provide your entire project in a zip file, so that we can diagnose your issue? We're unable to reproduce your error as you have not provided the whole code in the issue.

[harishsk]

Hi @adriansd27 If you still have this issue, can you please reopen the issue with a small sample that we can run and reproduce the issue?