Problem with runModelOnImage

[BoHellgren]

No description provided.

[BoHellgren]

I create a .png file from a Container widget. First I wrap the Container with a RepantBoundary widget:
child: RepaintBoundary( key: _globalKey, child: Container( width: MediaQuery.of(context).size.width, height: MediaQuery.of(context).size.width, color: Colors.lightBlue, child: Image.file(croppedF))))),

Then I run this code to create the png file:

`RenderRepaintBoundary boundary =
_globalKey.currentContext.findRenderObject();
ui.Image image = await boundary.toImage(pixelRatio: 3.0);
ByteData byteData =
await image.toByteData(format: ui.ImageByteFormat.png);
var buffer = byteData.buffer;
testFile = carPath + '/' + DateTime.now().toString() + '.png';
File(testFile).writeAsBytesSync(
buffer.asUint8List(byteData.offsetInBytes, byteData.lengthInBytes));
print('test file $testFile written');

`
The file is analyzed with Tensorflow Lite:

var recognitions = await Tflite.runModelOnImage( path: testFile, numResults: 5, threshold: 0.15, imageMean: 127.5, imageStd: 127.5, );

This works fine. But shouldn't it be possible to analyze the image directly, without creating a .png file, and instead use Tflite.runModelOnBinary? I cannot get this to work. What should the binary: parameter be? I can't find that this is documented. Could you please add instructions in the Readme file?

[BoHellgren]

.

[shaqian]

Hi BoHellgren,

README does have examples of how images should be converted to binary.

// for float model
Uint8List imageToByteListFloat32(
    img.Image image, int inputSize, double mean, double std) {
  var convertedBytes = Float32List(1 * inputSize * inputSize * 3);
  var buffer = Float32List.view(convertedBytes.buffer);
  int pixelIndex = 0;
  for (var i = 0; i < inputSize; i++) {
    for (var j = 0; j < inputSize; j++) {
      var pixel = image.getPixel(j, i);
      buffer[pixelIndex++] = (img.getRed(pixel) - mean) / std;
      buffer[pixelIndex++] = (img.getGreen(pixel) - mean) / std;
      buffer[pixelIndex++] = (img.getBlue(pixel) - mean) / std;
    }
  }
  return convertedBytes.buffer.asUint8List();
}

// for int model
Uint8List imageToByteListUint8(img.Image image, int inputSize) {
  var convertedBytes = Uint8List(1 * inputSize * inputSize * 3);
  var buffer = Uint8List.view(convertedBytes.buffer);
  int pixelIndex = 0;
  for (var i = 0; i < inputSize; i++) {
    for (var j = 0; j < inputSize; j++) {
      var pixel = image.getPixel(j, i);
      buffer[pixelIndex++] = img.getRed(pixel);
      buffer[pixelIndex++] = img.getGreen(pixel);
      buffer[pixelIndex++] = img.getBlue(pixel);
    }
  }
  return convertedBytes.buffer.asUint8List();
}

Or in the example app, the code is here
https://github.com/shaqian/flutter_tflite/blob/master/example/lib/main.dart#L173

  Future recognizeImageBinary(File image) async {
    var imageBytes = (await rootBundle.load(image.path)).buffer;
    img.Image oriImage = img.decodeJpg(imageBytes.asUint8List());
    img.Image resizedImage = img.copyResize(oriImage, 224, 224);
    var recognitions = await Tflite.runModelOnBinary(
      binary: imageToByteListFloat32(resizedImage, 224, 127.5, 127.5),
      numResults: 6,
      threshold: 0.05,
    );
    setState(() {
      _recognitions = recognitions;
    });
  }

runModelOnBinary is added in response to one user's request and thus it's implemented as the user requested it to be.

Thanks,
Qian

[shaqian]

I'm archiving this thread. Feel free to reopen if you have further questions.

Thanks,
Qian