Is there any way to pass pre-loaded image into MLContext pipeline?

[ma-gu]

System information

• OS version/distro: Windows 10.0.17134
• .NET Version (eg., dotnet --info): .NET Core 2.2.202

Issue

• What did you do?
  My intention was to implement object detection in video. Please see attached code sample and explanation below.
• What happened?
  I wasn't able to find proper API methods for configuring MLContext pipeline.
• What did you expect?
  I expected it should be possible to pass individual in-memory representation of image(as a Bitmap object for example) into pipeline.

Source code / logs

Let's pretend we have a task to run object detection model on frames coming from video stream.
It seems natural to pass 1 frame one by one as they arriving from camera.

I can't find such possibility with current MLContext pipeline and API available.

Here's example pipeline from machinelearning-samples :

            var pipeline = mlContext.Transforms.LoadImages(outputColumnName: "image", imageFolder: imagesFolder, inputColumnName: nameof(ImageNetData.ImagePath))
                            .Append(mlContext.Transforms.ResizeImages(outputColumnName: "image", imageWidth: ImageNetSettings.imageWidth, imageHeight: ImageNetSettings.imageHeight, inputColumnName: "image"))
                            .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
                            .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelLocation, outputColumnNames: new[] { TinyYoloModelSettings.ModelOutput }, inputColumnNames: new[] { TinyYoloModelSettings.ModelInput }));

Having LoadImages extension method overload accepting in-memory image representation (Bitmap or whatever) would definetly solve a problem:

public static ImageLoadingEstimator LoadImages(this TransformsCatalog catalog, params Bitmap[] images);

Am I missing some way implement this without storing frames onto the disk? Or does it mean ML.NET wasn't designed for realtime video processing?

Thanks in advance.

[ma-gu]

Basically it seems to be a duplicate of #3369 but re-worded.

[yaeldekel]

Hi @ma-gu ,
Currently, the only way to skip the LoadImages step in the pipeline and pass a Bitmap directly to the pipeline, is by adding a custom IDataView implementation, similar to the one in this sample:
https://github.com/dotnet/machinelearning/blob/master/docs/samples/Microsoft.ML.Samples/Dynamic/SimpleDataViewImplementation.cs#L116

The sample implementation creates an IDataView from an IEnumerable of InputObjects.

You can replace that with a class that has a public field Image of type Bitmap, and replace the getters array in line 197 with an array of length 1 containing a ValueGetter<Bitmap> that returns the Image field of the current object in the IEnumerable.

You can then start your pipeline with a ResizeImages estimator, and fit it using an instance of this custom IDataView class.

[ma-gu]

Hi @yaeldekel ,
Thanks for instructions, I've already found SimpleDataViewImplementation and simialr advice, but I got stuck with implementing Cursor. Will try your suggestion and get back.

Thanks.

[ma-gu]

Hi @yaeldekel,

I've implemented BitmapDataView, could you please to have a look if you spot anything obviously wrong?

An issue I have currently is ML.NET throwing the following exception during prediction engine construction:

Could not determine an IDataView type for member Schema
Parameter name: rawType

   at Microsoft.ML.Data.InternalSchemaDefinition.GetVectorAndItemType(Type rawType, String name, Boolean& isVector, Type& itemType)
   at Microsoft.ML.Data.SchemaDefinition.Create(Type userType, Direction direction)
   at Microsoft.ML.Data.DataViewConstructionUtils.CreateInputRow[TRow](IHostEnvironment env, SchemaDefinition schemaDefinition)
   at Microsoft.ML.PredictionEngineBase`2..ctor(IHostEnvironment env, ITransformer transformer, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at Microsoft.ML.PredictionEngineExtensions.CreatePredictionEngine[TSrc,TDst](ITransformer transformer, IHostEnvironment env, Boolean ignoreMissingColumns, SchemaDefinition inputSchemaDefinition, SchemaDefinition outputSchemaDefinition)
   at ObjectDetection.OnnxModelScorer.LoadModel(BitmapDataView imageData) in C:\Users\Maxim\Source\Repos\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ObjectDetection_Onnx_Visualization\ObjectDetection.Core\OnnxModelScorer.cs:line 70
   at ObjectDetection.OnnxModelScorer.Score(Bitmap image) in C:\Users\Maxim\Source\Repos\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ObjectDetection_Onnx_Visualization\ObjectDetection.Core\OnnxModelScorer.cs:line 49
   at TestConsole.Program.Main(String[] args) in C:\Users\Maxim\Source\Repos\machinelearning-samples\samples\csharp\getting-started\DeepLearning_ObjectDetection_Onnx_Visualization\TestConsole\Program.cs:line 21

Exception is throwing from the last line in the following snippet(linked above to code in my fork of machinelearning-samples):

    var imageData = new BitmapDataView(image);
    var pipeline = mlContext.Transforms.ResizeImages(outputColumnName: "image", imageWidth: ImageNetSettings.imageWidth, imageHeight: ImageNetSettings.imageHeight, inputColumnName: "image")
       .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
       .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelLocation, outputColumnNames: new[] { TinyYoloModelSettings.ModelOutput }, inputColumnNames: new[] { TinyYoloModelSettings.ModelInput }));
    var model = pipeline.Fit(imageData);
    var predictionEngine = mlContext.Model.CreatePredictionEngine<BitmapDataView, ImageNetPrediction>(model);

Unfortunately I stuck at this point and seems Exception messages from ML.NET doesn't look helpful for those who isn't familiar with this library. I have a guess that something is wrong with my BitmapDataView implementation but can't figure out what exactly.

Thanks

[codemzs]

@wschin Does your fix address this?

[jamsoft]

@ma-gu Did you ever get this working?

[ma-gu]

@jamsoft Almost 2 years passed already... I believe I just gave up - otherwise I should provide working code snippet here.
I tired fighting with ML.NET writing unnecessary code in attempts to make it working when the same feature lives as one-liner in python stack(keras/tensorflow).

[jamsoft]

Thanks for your reply. Yeah, it's cost me a day to find out no-one can get it working and it seems like the most obvious use case as well. So odd. I also cannot use my 3090 with ML.NET as it needs drivers from 2019. I think I'm going to go look at bundling a Python exe again.

[bbday]

Also me have same issue with image processing, need to look into python trainer to avoid this issue.
In my case i need to rotate images at runtime during training to avoid to create X copies on my hdd.