Prediction engine outputs only one class

[elisaho]

System Information (please complete the following information):

• Windows 10 build 19044.1766
• ML.NET v1.7.1
• .NET 6.0

Describe the bug
For a given image input to the prediction engine, the output schema only allows for one output detected class (and a single corresponding score). However, I would like for all detected classes to be returned.
I am working on consuming the tensorflow model for object detection (exported from Custom Vision as GeneralCompactS1) in .Net using ML.NET. I have closely referenced:
https://github.com/dotnet/machinelearning-samples/tree/main/samples/csharp/end-to-end-apps/DeepLearning_ImageClassification_TF/TFImageClassification
I know the model can return multiple detected classes as I have consumed the same tensorflow model with python, tensorflowsharp, and tensorflow.net. I have tried appending column names with ':0' and ':1'(e.g. detected_classes:0) and changing the output tensor types (e.g. for detected_classes, change long to long[]), both to no avail.

Expected behavior
I expect all detected classes and their corresponding scores to be returned.

Screenshots, Code, Sample Projects

[michaelgsharp]

Would you be able to share sample code/data and especially the model so we can take a look?

This issue has been marked needs-author-action and may be missing some important information.

[elisaho]

Hi, thanks for getting back to me! Unfortunately I cannot share my model but you can generate a sample model of the same kind by training multiple object detected classes (e.g. orange, apple, pear) in Azure's custom vision with Object Detection General Compact s1.

[michaelgsharp]

We are looking into generating one so that we can investigate this further. While we are doing that, have you tried running this with addBatchDimensionInput set to true? I'm wondering if it has to do with the output shape are how we parse it.

[elisaho]

Running with addBatchDimensionsInput set to true yields the exception InvalidArgumentError: The first dimension of paddings must be the rank of inputs[4,2] [1,1,320,320,3]. I have moved away from using the tensorflow to using an ONNX model instead. Thanks for your help though!

[luisquintanilla]

@elisaho just want to confirm, for the export format, you were using TensorFlow? If so, which TF format?

[elisaho]

@luisquintanilla