How to use multiple Output Tensors?

[alexw92]

I have an OD model which 4 output tensors. However I am unsure how should I give them to the loaded interpreter.
Lets take a look at the code of interpreter.dart

void run(Object input, Object output) {
    var map = <int, Object>{};
    map[0] = output;
    runForMultipleInputs([input], map);
  }

  /// Run for multiple inputs and outputs
  void runForMultipleInputs(List<Object> inputs, Map<int, Object> outputs) {
    if (outputs.isEmpty) {
      throw ArgumentError('Input error: Outputs should not be null or empty.');
    }
    runInference(inputs);
    var outputTensors = getOutputTensors();
    for (var i = 0; i < outputTensors.length; i++) {
      outputTensors[i].copyTo(outputs[i]!);
    }
  }

So we can see that the output fed into model is assigned to the first field of a Map<int, Object>. So then in runForMultipleInputs in the for-loop this outputTensors[i].copyTo(outputs[i]!) runs into a null access exception.
This is how I feed the output to the model.

    final output0 = List<List<double>>.filled(1, List<double>.filled(25, 0));
    final output1 = List<List<List<double>>>.filled(1, List<List<double>>.filled(25, List<double>.filled(4, 0)));
    final output2 = List<double>.filled(1, 0);
    final output3 = List<List<double>>.filled(1, List<double>.filled(25, 0));
    Map<int, Object> output = {
      0: output0,
      1: output1,
      2: output2,
      3: output3};
    print(output);
    interpreter.run(image,output );

So far I only made it working by just casting the input object to Map. How is this supposed to be used anyway?

  void run(Object input, Object output) {
    var map = <int, Object>{};
    map[0] = output;
    runForMultipleInputs([input], output as Map<int, Object> );
  }

[gregorscholz]

I will hopefully push soon an example for object detection where the model takes one input tensor and has 4 output tensors.
Hopefully i understand you correctly and this helps you.

// Set input tensor [1, 300, 300, 3]
final input = [imageMatrix];

// Set output tensor
// Locations: [1, 10, 4]
// Classes: [1, 10],
// Scores: [1, 10],
// Number of detections: [1]
final output = {
  0: [List<List<num>>.filled(10, List<num>.filled(4, 0))],
  1: [List<num>.filled(10, 0)],
  2: [List<num>.filled(10, 0)],
  3: [0.0],
};

 _interpreter.runForMultipleInputs([input], output);

[alexw92]

Yes thanks this is what I ended up doing. At first I didnt realize that runForMultipleInputs is directly callable from outside too and was using run only. By directly using runForMultipleInputs this works perfectly for OD. Just the naming could be improved. Thank you!

[gregorscholz]

Same for me. Thats why i opened this ticket here.

[Rikerz08]

Hi @gregorscholz and @alexw92! I'm fairly new to ML and flutter development, do you know how to solve this error? I have a MobileNetSSD 320x320 model.

Exception has occurred.
ArgumentError (Invalid argument(s): Output object shape mismatch, interpreter returned output of shape: [1, 10] while shape of output provided as argument in run is: [1, 10, 4])

[alexw92]

@Rikerz08 Hello there, you need to specify outputs which fit the dimensions of the outputs ur model uses. You can find out the output dims of your model by using Netron or something similiar

[Rikerz08]

@alexw92 Thanks for this! However, I'm confused as to why I still need to determine my model's output shape. Isn't it [1,10] as what is stated in the error? Please correct me if I'm wrong.

Additionally, how do I specify and change the expected output size in order to fit my model's shape in this code?

import 'dart:developer';
import 'dart:io';
import 'package:flutter/services.dart';
import 'package:image/image.dart' as img;
import 'package:tflite_flutter/tflite_flutter.dart';

class ObjectDetection {
  static const String _modelPath = 'assets/models/Hemalens.tflite';
  static const String _labelPath = 'assets/models/labelmap.txt';

  Interpreter? _interpreter;
  List<String>? _labels;

  ObjectDetection() {
    _loadModel();
    _loadLabels();
    log('Done.');
  }

  Future<void> _loadModel() async {
    log('Loading interpreter options...');
    final interpreterOptions = InterpreterOptions();

    // Use XNNPACK Delegate
    if (Platform.isAndroid) {
      interpreterOptions.addDelegate(XNNPackDelegate());
    }

    // Use Metal Delegate
    if (Platform.isIOS) {
      interpreterOptions.addDelegate(GpuDelegate());
    }

    log('Loading interpreter...');
    _interpreter =
        await Interpreter.fromAsset(_modelPath, options: interpreterOptions);
  }

  Future<void> _loadLabels() async {
    log('Loading labels...');
    final labelsRaw = await rootBundle.loadString(_labelPath);
    _labels = labelsRaw.split('\n');
  }

  Uint8List analyseImage(String imagePath) {
    log('Analysing image...');
    // Reading image bytes from file
    final imageData = File(imagePath).readAsBytesSync();

    // Decoding image
    final image = img.decodeImage(imageData);

    // Resizing image fpr model, [300, 300]
    final imageInput = img.copyResize(
      image!,
      width: 300,
      height: 300,
    );

    // Creating matrix representation, [300, 300, 3]
    final imageMatrix = List.generate(
      imageInput.height,
      (y) => List.generate(
        imageInput.width,
        (x) {
          final pixel = imageInput.getPixel(x, y);
          return [pixel.r, pixel.g, pixel.b];
        },
      ),
    );

    final output = _runInference(imageMatrix);

    log('Processing outputs...');
    // Location
    final locationsRaw = output.first.first as List<List<double>>;
    final locations = locationsRaw.map((list) {
      return list.map((value) => (value * 300).toInt()).toList();
    }).toList();
    log('Locations: $locations');

    // Classes
    final classesRaw = output.elementAt(1).first as List<double>;
    final classes = classesRaw.map((value) => value.toInt()).toList();
    log('Classes: $classes');

    // Scores
    final scores = output.elementAt(2).first as List<double>;
    log('Scores: $scores');

    // Number of detections
    final numberOfDetectionsRaw = output.last.first as double;
    final numberOfDetections = numberOfDetectionsRaw.toInt();
    log('Number of detections: $numberOfDetections');

    log('Classifying detected objects...');
    final List<String> classication = [];
    for (var i = 0; i < numberOfDetections; i++) {
      classication.add(_labels![classes[i]]);
    }

    log('Outlining objects...');
    for (var i = 0; i < numberOfDetections; i++) {
      if (scores[i] > 0.6) {
        // Rectangle drawing
        img.drawRect(
          imageInput,
          x1: locations[i][1],
          y1: locations[i][0],
          x2: locations[i][3],
          y2: locations[i][2],
          color: img.ColorRgb8(255, 0, 0),
          thickness: 3,
        );

        // Label drawing
        img.drawString(
          imageInput,
          '${classication[i]} ${scores[i]}',
          font: img.arial14,
          x: locations[i][1] + 1,
          y: locations[i][0] + 1,
          color: img.ColorRgb8(255, 0, 0),
        );
      }
    }

    log('Done.');

    return img.encodeJpg(imageInput);
  }

  List<List<Object>> _runInference(
    List<List<List<num>>> imageMatrix,
  ) {
    log('Running inference...');

    // Set input tensor [1, 300, 300, 3]
    final input = [imageMatrix];

    // Set output tensor
    // Locations: [1, 10, 4]
    // Classes: [1, 10],
    // Scores: [1, 10],
    // Number of detections: [1]
    final output = {
      0: [List<List<num>>.filled(10, List<num>.filled(4, 0))],
      1: [List<num>.filled(10, 0)],
      2: [List<num>.filled(10, 0)],
      3: [0.0],
    };

    _interpreter!.runForMultipleInputs([input], output);
    return output.values.toList();
  }
}

[alexw92]

@Rikerz08 The error you see is probably a mismatch of one of the 4 outputs assuming your model also has 4 outputs. Seeing from the code and the error I would guess that the order of the outputs is different in your model. What happens if you change this

final output = {
      0: [List<List<num>>.filled(10, List<num>.filled(4, 0))],
      1: [List<num>.filled(10, 0)],
      2: [List<num>.filled(10, 0)],
      3: [0.0],
    };

to this

final output = {
      0: [List<num>.filled(10, 0)],
      1: [List<List<num>>.filled(10, List<num>.filled(4, 0))],
      2: [List<num>.filled(10, 0)],
      3: [0.0],
    };

for instance?

[Rikerz08]

@alexw92

We tried changing it to this:

final output = { 0: [List<num>.filled(10, 0)], 1: [List<List<num>>.filled(10, List<num>.filled(4, 0))], 2: [0.0], 3: [List<num>.filled(10, 0)], };

The shape error was fixed however it had another error which is this one:

_TypeError (type 'List<double>' is not a subtype of type 'List<List<double>>' in type cast)

[alexw92]

@Rikerz08 ah so thats some progress. Just try out all the permutations and in the end you will have the right order. Or add a print statement in the runForMultipleInputs code to check the dimensions of each output manually. Or you use Netron to look up the outputs of your model.

[LachhabMeriem]

E/flutter (30995): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Output object shape mismatch, interpreter returned output of shape: [1, 84, 8400] while shape of output provided as argument in run is: [8400, 4]