Camera document how to use ImageStream

[eduardkieser]

First of all thank you so mush for adding access to the image stream in 0.2.8, I'm sure may people will be very happy about this.
I know the latest commit is fresh off the press, bit I would love to see a little more documentation on how to use the camera image stream (other than: "use: cameraController.startImageStream(listener) to process the images")

[lewixlabs]

Yes agree, great feature but it needs documentation.
It seems working, i added this new method and its callback in initState inside demo project

  @override
  void initState() {
    super.initState();
    controller = CameraController(cameras[0], ResolutionPreset.medium);
    controller.initialize().then((_) {
      if (!mounted) {
        return;
      }

      controller.startImageStream((CameraImage availableImage) {
        controller.stopImageStream();
        _scanText(availableImage);
      });

      setState(() {});
    });
  }

void _scanText(CameraImage availableImage) async {
  
  final FirebaseVisionImage visionImage = FirebaseVisionImage.fromBytes(availableImage.planes[0].bytes, null);
  final TextRecognizer textRecognizer = FirebaseVision.instance.textRecognizer();
  final VisionText visionText = await textRecognizer.processImage(visionImage);

  print(visionText.text);
  for (TextBlock block in visionText.blocks) {

    // final Rectangle<int> boundingBox = block.boundingBox;
    // final List<Point<int>> cornerPoints = block.cornerPoints;
    print(block.text);
    final List<RecognizedLanguage> languages = block.recognizedLanguages;

    for (TextLine line in block.lines) {
      // Same getters as TextBlock
      print(line.text);
      for (TextElement element in line.elements) {
        // Same getters as TextBlock
        print(element.text);
      }
    }
  }
}

Now i'm learning how to handle method FirebaseVisionImage.fromBytes to set metadata

[lewixlabs]

This is a code sample just update
https://github.com/lewixlabs/mlkit_ocr_realtime/blob/96a749c0568247fc0063d52a4f03f0c6385f52cc/lib/main.dart#L118-L168
but not working because i get this error

PlatformException (PlatformException(textRecognizerError, Internal error has occurred when executing Firebase ML tasks, null))

when

final VisionText visionText = await textRecognizer.processImage(visionImage);

is called.
Full sample is here
https://github.com/lewixlabs/mlkit_ocr_realtime/blob/master/lib/main.dart

Waiting for the right direction, thank you! :)

[lewixlabs]

Uuuuh, my updated code works on Android! :)
Tomorrow i test it on iOS...

  void _scanText(CameraImage availableImage) async {
    _isScanBusy = true;

    print("scanning!...");

    /*
     * https://firebase.google.com/docs/ml-kit/android/recognize-text
     * .setWidth(480)   // 480x360 is typically sufficient for
     * .setHeight(360)  // image recognition
     */

    final FirebaseVisionImageMetadata metadata = FirebaseVisionImageMetadata(
      rawFormat: availableImage.format.raw,
      size: Size(availableImage.width.toDouble(),availableImage.height.toDouble()),
      planeData: availableImage.planes.map((currentPlane) => FirebaseVisionImagePlaneMetadata(
        bytesPerRow: currentPlane.bytesPerRow,
        height: currentPlane.height,
        width: currentPlane.width
        )).toList(),
      rotation: ImageRotation.rotation90
      );

    final FirebaseVisionImage visionImage = FirebaseVisionImage.fromBytes(availableImage.planes[0].bytes, metadata);
    final TextRecognizer textRecognizer = FirebaseVision.instance.textRecognizer();
    final VisionText visionText = await textRecognizer.processImage(visionImage);

    print("--------------------visionText:${visionText.text}");
    for (TextBlock block in visionText.blocks) {
      // final Rectangle<int> boundingBox = block.boundingBox;
      // final List<Point<int>> cornerPoints = block.cornerPoints;
      print(block.text);
      final List<RecognizedLanguage> languages = block.recognizedLanguages;

      for (TextLine line in block.lines) {
        // Same getters as TextBlock
        print(line.text);
        for (TextElement element in line.elements) {
          // Same getters as TextBlock
          print(element.text);
        }
      }
    }

    _isScanBusy = false;
  }

Full demo project here
https://github.com/lewixlabs/mlkit_ocr_realtime

[BoHellgren]

I am trying to display an image from an ImageStream. One one button I have

onPressed: () async  {
                await controller.startImageStream((CameraImage availableImage)  {
                  print(availableImage.planes[0].bytes.toString().substring(0,200));
                  setState(() {
                    theImage = availableImage.planes[0].bytes;
                  });
                });

where theImage is declared as Uint8List.

A list of numbers between 0 and 255 is displayed. Like

I/flutter (18471): [16, 118, 115, 112, 114, 117, 118, 119, 120, 120, 120, 120, 118, 120, etc

But if I try

Image.memory(theImage) or
MemoryImage(theImage) I get

[ERROR:flutter/lib/ui/painting/codec.cc(97)] Failed decoding image. Data is either invalid, or it is encoded using an unsupported format.

What am I doing wrong?

(I also have great problems stopping the ImageStream with controller.stopImageStream() ).

[zoechi]

@lewixlabs @eduardkieser Using "add reaction" on the initial comment would increase priority while +1 comments are rather pointless and cause lots of people to get notified for no reason.
To get notified yourself use the [Subscribe] button to the top right instead.

[BoHellgren]

This is a sample of what the CameraImage contains:

I/flutter (27030): Format group: ImageFormatGroup.yuv420
I/flutter (27030): height: 480
I/flutter (27030): width: 640
I/flutter (27030): Number of planes: 3
I/flutter (27030): plane 0 length: 307200
I/flutter (27030): plane 0 data: [16, 21, 21, 19, 19,
I/flutter (27030): plane 1 length: 76800
I/flutter (27030): plane 1 data: [130, 126, 126, 125,
I/flutter (27030): plane 2 length: 76800
I/flutter (27030): plane 2 data: [125, 131, 131, 132,

How can I convert this to a Uint8List acceptable to Image.memory()??
Or is there another widget which can display a CameraImage?

[rockingelevator]

here is a great demo of how to use image stream and face detection: https://github.com/bparrishMines/mlkit_demo/blob/master/lib/face_expression_reader.dart

[lewixlabs]

here is a great demo of how to use image stream and face detection: https://github.com/bparrishMines/mlkit_demo/blob/master/lib/face_expression_reader.dart

Uh, interesting, i think is one of the authors of the official plugin.
Thanks, i will take a look

[sathiez]

How to stream Images with correct number of pixels

controller.startImageStream((CameraImage availableImage){
    var framesY = availableImage.planes[0].bytes;
    var framesU = availableImage.planes[1].bytes;
    var framesV = availableImage.planes[2].bytes;
    var lenU = framesU.length;
    var lenV = framesV.length;
    var lenY = framesY.length;
    print('$lenU'+' '+'$lenV'+' '+'$lenY');
  });

where,

lenU = 460799
lenV = 460799
lenY = 921600

I con't convert YUV_420 to BGR. I hope lenU & lenV should be 460800, if not, can you suggest me best way to convert YUV_240 to BGR.

[lewixlabs]

here is a great demo of how to use image stream and face detection: https://github.com/bparrishMines/mlkit_demo/blob/master/lib/face_expression_reader.dart

Link today is dead
Main project can be found here
https://github.com/bparrishMines/mlkit_demo/

Demo code is here
https://github.com/bparrishMines/mlkit_demo/blob/master/lib/main.dart

[sathiez]

why do I get U & V like
lenY = 921600 lenU = 460799 lenV = 460799
which is closer to u=Y/2, but I need to get u & v= Y/4 because it is YUV420. any suggestion on this?

[sathiez]

#26348 (comment) any suggestion on this?

[BoHellgren]

@sathiez How do you convert yuv420 to BGR (if you have valid input)? And can the result be used in Image.memory()?

(I think it is a very reasonable scenario to scan images from an ImageStream, and, when an image meets certain criteria, display it with Image.memory, or some other widget. But I haven't been able to do that.)

[sathiez]

I tried to convert in python using opencv
BGR = cv2.cvtColor(YUV, cv2.COLOR_YUV2BGR_NV21)

[alejamp]

Same problem here, I need to display in a widget the CameraImage YUV420_888.
I wrote a piece of code... it's horrible, very slow, not efficient at all. But It was the closest thing I got from seeing a YUV420 image on the screen.

      // _image -> YUV420_888 CameraImage from Camera Plugin startImageStream
      // Create Image buffer
      // imgLib -> Image package from https://pub.dartlang.org/packages/image
      var img = imglib.Image(width, height);

      // Fill image buffer with plane[0] from YUV420_888
      for(int x=0; x < width; x++) {
        for(int y=0; y < height; y++) {
          final pixelColor = _image.planes[0].bytes[y * width + x];
          // color: 0x FF  FF  FF  FF 
          //           A   B   G   R
          // Calculate Grayscale pixel 
          img.data[y * width + x] = (0xFF << 24) | (pixelColor << 16) | (pixelColor << 8) | pixelColor;
        }
      }

      List<int> png = imglib.encodePng(img);
      return Image.memory(png);

At least I can see the black and white image rotated 90 degrees on screen.
Samsung S9+ requieres up to 600ms to perform this task, so it's no suitable for real time.

[alejamp]

Disabling compression of the PNG encoder, and setting filter to none, improves performance a lot

  // CameraImage YUV420_888 -> PNG -> Image (compresion:0, filter: none)
  // Black
  Future<Image> convertYUV420toImage(CameraImage image) async {

      try {
        final int width = image.width;
        final int height = image.height;

        
        // imgLib -> Image package from https://pub.dartlang.org/packages/image
        var img = imglib.Image(width, height); // Create Image buffer

        // Fill image buffer with plane[0] from YUV420_888
        for(int x=0; x < width; x++) {
          for(int y=0; y < height; y++) {
            final pixelColor = image.planes[0].bytes[y * width + x];
            // color: 0x FF  FF  FF  FF 
            //           A   B   G   R
            // Calculate pixel color
            img.data[y * width + x] = (0xFF << 24) | (pixelColor << 16) | (pixelColor << 8) | pixelColor;
          }
        }

        imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
        List<int> png = pngEncoder.encodeImage(img);
        muteYUVProcessing = false;
        return Image.memory(png);  
      } catch (e) {
        print(">>>>>>>>>>>> ERROR:" + e.toString());
      }
      return null;
  }

[sathiez]

How to compress image stream efficiently so that I can send it through websocket?

[alejamp]

How to compress image stream efficiently so that I can send it through websocket?

Level parameter is compresion level. Increase that value. Default is 6. And don't change default filter

imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 9);

[BoHellgren]

@alejamp Thank you so much for this code, it works great in my app, with OK performance.
But the image is black-and-white (grayscale). Which could be expected when you only use plane 0.
Do you know how to get the image in color?

[CellCS]

@sikandernoori for ImageFormatGroup.bgra8888 in ios, now uses ResolutionPreset.veryHigh.

But still crash in ipad mini 2 with ios 12 when convert into RGB (native codes) and encodeJpg later.

[sikandernoori]

@CellCS I found the Solution for it.

@alexcohn mentioned it somewhere. Solution was as below:

Future<imglib.Image?> _convertBGRA8888(CameraImage image) async {
  try {
    return imglib.Image.fromBytes(
      image.planes[0].bytesPerRow ~/ 4,
      image.height,
      image.planes[0].bytes,
      format: imglib.Format.bgra,
    );
  } catch (ex) {
    return null;
  }
}

[SebghatYusuf]

@sikandernoori
I've tried using your code snippet, but it's not working fine with iOS 15, it converts the image, the width of the image is not wide as it should be and the image is in grayscale, can't convert to a color image.
any suggestions?

[sikandernoori]

@SebghatYusuf Can You Please share Meta Information of CameraImage and what resolution preset you are using.

[SebghatYusuf]

@sikandernoori
Thank you so much, resolved the issue.

[physxP]

@SebghatYusuf can you please elaborate on how you solved the issue? I am still getting grayscale image from @sikandernoori ' s method

[SebghatYusuf]

@physxP
Sure, I'm using the CameraController to take pictures instead of converting the YUV420/BGRA8888 image to png/jpeg

 if (Platform.isIOS) {
      _cameraController!.stopImageStream();
      var capturedImage = await _cameraController!.takePicture();
      widget.setCapturedImage(capturedImage);
      return;
    }

[gaburielcasado]

To anyone looking into using compute to perform the conversion on an isolate:
UInt8List sometimes breaks compute for some reason that I've not been able to discover, but converting it to List "solves" the issue. Take note that List is less efficient than UInt8List.

[lala-naibova]

This code working in ios very well, but cannot read data in android, don't know which conversion I need to use for taking data from cameraImage

@override
  void initState() {
    //onCameraSelected(widget.cameras[0]);
    super.initState();

    controller = CameraController(widget.cameras[0], ResolutionPreset.medium,
        enableAudio: false);

    controller.initialize().then((_) {
      if (!mounted) {
        return;
      }

      setState(() {});
      _timer = Timer.periodic(Duration(seconds: 4), (currentTimer) async {
        await controller.startImageStream((CameraImage availableImage) async {
          //controller.stopImageStream();
          _scanText(availableImage);
        });
      });
    });
  }

  void _scanText(CameraImage availableImage) async {
    try {
      if (_isScanBusy) return;
      setState(() {
        _isScanBusy = true;
      });

      final WriteBuffer allBytes = WriteBuffer();
      for (Plane plane in availableImage.planes) {
        allBytes.putUint8List(plane.bytes);
      }
      final bytes = allBytes.done().buffer.asUint8List();

      final Size imageSize = Size(
          availableImage.width.toDouble(), availableImage.height.toDouble());

      final InputImageRotation imageRotation =
          InputImageRotationMethods.fromRawValue(
                  widget.cameras[0].sensorOrientation) ??
              InputImageRotation.Rotation_0deg;

      final InputImageFormat inputImageFormat =
          InputImageFormatMethods.fromRawValue(availableImage.format.raw) ??
              InputImageFormat.NV21;

      final planeData = availableImage.planes.map(
        (Plane plane) {
          return InputImagePlaneMetadata(
            bytesPerRow: plane.bytesPerRow,
            height: plane.height,
            width: plane.width,
          );
        },
      ).toList();

      final inputImageData = InputImageData(
        size: imageSize,
        imageRotation: imageRotation,
        inputImageFormat: inputImageFormat,
        planeData: planeData,
      );

      final inputImage =
          InputImage.fromBytes(bytes: bytes, inputImageData: inputImageData);

      final textDetector = GoogleMlKit.vision.textDetector();
      final RecognisedText recognisedText =
          await textDetector.processImage(inputImage);
      print('Texttttt ======= ${recognisedText.text}');
    
      _isScanBusy = false;
    } catch (e) {
      print(e);
    }
  }

[KingWu]

Any example demo how convert to image and then display it. i tried the the convert function. Display the image from Image.memory(png). Not work.

SafeArea(
    child:Scaffold(
      body: SizedBox.expand(
        child: Container(
          color: Colors.red,
          child: image,
        ),
      ),
    )),

[Anfet]

Don't know if this is will help, I've made a small dart plugin to help manipulate these camera images. Then they can be safely passed to detectors.

https://github.com/Anfet/flutter-yuvimage

[BoHellgren]

CameraImage to imageLib.Image performance
I made the function below which converts a CameraImage to a imageLib.Image using a) C code and Dart.ffi, and b) plain Dart. I expected the C code to be much faster, but, to my surprise, the Dart code is some 25% faster! Can anyone explain why? Any ideas on how to do this conversion really fast?

import 'dart:typed_data';
import 'package:ffi/ffi.dart';
import 'package:image/image.dart' as imageLib;
import 'package:camera/camera.dart';

typedef convert_func = Pointer<Uint32> Function(
    Pointer<Uint8>, Pointer<Uint8>, Pointer<Uint8>, Int32, Int32, Int32, Int32);
typedef Convert = Pointer<Uint32> Function(
    Pointer<Uint8>, Pointer<Uint8>, Pointer<Uint8>, int, int, int, int);

imageLib.Image convertCameraImage(CameraImage image, Convert conv) {
//*********************FFI VERSION******************************
  /* Put this code in main.dart
  final DynamicLibrary convertImageLib =
      DynamicLibrary.open("libconvertImage.so");
  Convert? conv;
  ... and in initState
  conv = convertImageLib
      .lookup<NativeFunction<convert_func>>('convertImage')
      .asFunction<Convert>(); */
  var t1 = DateTime.now().millisecondsSinceEpoch;
  // Allocate memory for the 3 planes of the image
  Pointer<Uint8> p = malloc.allocate(image.planes[0].bytes.length);
  Pointer<Uint8> p1 = malloc.allocate(image.planes[1].bytes.length);
  Pointer<Uint8> p2 = malloc.allocate(image.planes[2].bytes.length);
  // Assign the planes data to the pointers of the image
  Uint8List pointerList = p.asTypedList(image.planes[0].bytes.length);
  Uint8List pointerList1 = p1.asTypedList(image.planes[1].bytes.length);
  Uint8List pointerList2 = p2.asTypedList(image.planes[2].bytes.length);
  pointerList.setRange(0, image.planes[0].bytes.length, image.planes[0].bytes);
  pointerList1.setRange(0, image.planes[1].bytes.length, image.planes[1].bytes);
  pointerList2.setRange(0, image.planes[2].bytes.length, image.planes[2].bytes);

  // Call the convertImage function and convert the YUV to RGB
  Pointer<Uint32> imgP = conv(
      p,
      p1,
      p2,
      image.planes[1].bytesPerRow,
      image.planes[1].bytesPerPixel!,
      image.planes[0].bytesPerRow,
      image.height);

  // Get the pointer of the data returned from the function to a List
  List imgData = imgP.asTypedList((image.planes[0].bytesPerRow * image.height));
  // Generate image from the converted data
  imageLib.Image img1 = imageLib.Image.fromBytes(
      image.height, image.planes[0].bytesPerRow, imgData as List<int>);
  // Free the memory space allocated
  // from the planes and the converted data
  malloc.free(p);
  malloc.free(p1);
  malloc.free(p2);
  malloc.free(imgP);

  var t2 = DateTime.now().millisecondsSinceEpoch;
  // return img1;  //Comment to use Dart
//******************DART VERSION***********************
  var t3 = DateTime.now().millisecondsSinceEpoch;
  int width = image.width;
  int height = image.height;
  var img2 = imageLib.Image(width, height); // Create Image buffer
  const int hexFF = 0xFF000000;
  final int uvyButtonStride = image.planes[1].bytesPerRow;
  final int? uvPixelStride = image.planes[1].bytesPerPixel;
  for (int x = 0; x < width; x++) {
    for (int y = 0; y < height; y++) {
      final int uvIndex = uvPixelStride! * (x / 2).floor() +
          uvyButtonStride * (y / 2).floor();
      final int index = y * width + x;
      final yp = image.planes[0].bytes[index];
      final up = image.planes[1].bytes[uvIndex];
      final vp = image.planes[2].bytes[uvIndex];
// Calculate pixel color
      int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
      int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
          .round()
          .clamp(0, 255);
      int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
// color: 0x FF  FF  FF  FF
//           A   B   G   R
      img2.data[index] = hexFF | (b << 16) | (g << 8) | r;
    }
  }
// Rotate 90 degrees to upright
  var img3 = imageLib.copyRotate(img2, 90);
  var t4 = DateTime.now().millisecondsSinceEpoch;
  print('*********** conv took ${t2 - t1} milliseconds, Dart took ${t4 - t3} milliseconds');

  return img3;

}

Here is the C code (by @Hugand)

#include <stdio.h>
#include "converter.h"
#include <math.h>
#include <stdlib.h>

int clamp(int lower, int higher, int val){
    if(val < lower)
        return 0;
    else if(val > higher)
        return 255;
    else
        return val;
}

int getRotatedImageByteIndex(int x, int y, int rotatedImageWidth){
    return rotatedImageWidth*(y+1)-(x+1);
}

uint32_t *convertImage(uint8_t *plane0, uint8_t *plane1, uint8_t *plane2, int bytesPerRow, int bytesPerPixel, int width, int height){
     int hexFF = 255;
    int x, y, uvIndex, index;
    int yp, up, vp;
    int r, g, b;
    int rt, gt, bt;

    uint32_t *image = malloc(sizeof(uint32_t) * (width * height));

    for(x = 0; x < width; x++){
        for(y = 0; y < height; y++){
            
            uvIndex = bytesPerPixel * ((int) floor(x/2)) + bytesPerRow * ((int) floor(y/2));
            index = y*width+x;

            yp = plane0[index];
            up = plane1[uvIndex];
            vp = plane2[uvIndex];
            rt = round(yp + vp * 1436 / 1024 - 179);
            gt = round(yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91);
            bt = round(yp + up * 1814 / 1024 - 227);
            r = clamp(0, 255, rt);
            g = clamp(0, 255, gt);
            b = clamp(0, 255, bt);
            image[getRotatedImageByteIndex(y, x, height)] = (hexFF << 24) | (b << 16) | (g << 8) | r;
        }
    }
    return image;
}

[faslurrajah]

This works for me.

Future<Uint8List> convertImageToPng(CameraImage image) async {
    Uint8List bytes;
    try {
      imglib.Image img;
      if (image.format.group == ImageFormatGroup.yuv420) {
        bytes = await convertYUV420toImageColor(image);
      } else if (image.format.group == ImageFormatGroup.bgra8888) {
        img = _convertBGRA8888(image);
        imglib.PngEncoder pngEncoder = new imglib.PngEncoder();
        bytes = pngEncoder.encodeImage(img);
      }
      return bytes;
    } catch (e) {
      print(">>>>>>>>>>>> ERROR:" + e.toString());
    }
    return null;
  }
  imglib.Image _convertBGRA8888(CameraImage image) {
    return imglib.Image.fromBytes(
      image.width,
      image.height,
      image.planes[0].bytes,
      format: imglib.Format.bgra,
    );
  }
  Future<Uint8List> convertYUV420toImageColor(CameraImage image) async {
    try {
      final int width = image.width;
      final int height = image.height;
      final int uvRowStride = image.planes[1].bytesPerRow;
      final int uvPixelStride = image.planes[1].bytesPerPixel;
      print("uvRowStride: " + uvRowStride.toString());
      print("uvPixelStride: " + uvPixelStride.toString());
      var img = imglib.Image(width, height); // Create Image buffer
      // Fill image buffer with plane[0] from YUV420_888
      for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
          final int uvIndex =
              uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
          final int index = y * width + x;
          final yp = image.planes[0].bytes[index];
          final up = image.planes[1].bytes[uvIndex];
          final vp = image.planes[2].bytes[uvIndex];
          int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
          int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
              .round()
              .clamp(0, 255);
          int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
          img.data[index] = (0xFF << 24) | (b << 16) | (g << 8) | r;
        }
      }
      imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
      Uint8List png = pngEncoder.encodeImage(img);
      final originalImage = imglib.decodeImage(png);
      final height1 = originalImage.height;
      final width1 = originalImage.width;
      imglib.Image fixedImage;
      if (height1 < width1) {
        fixedImage = imglib.copyRotate(originalImage, 90);
      }
      final path =
          join((await getTemporaryDirectory()).path, "${DateTime.now()}.jpg");
      print(path);
      File(path).writeAsBytesSync(imglib.encodeJpg(fixedImage));
      return imglib.encodeJpg(fixedImage);
    } catch (e) {
      print(">>>>>>>>>>>> ERROR:" + e.toString());
    }
    return null;
  }

I'm trying to the same. But I cannot edit ImageData variable's index. im1.data is a ImageData type. I'm trying to do as you do img.data[index] = (0xFF << 24) | (b << 16) | (g << 8) | r;

I'm getting compiling error The operator '[]=' isn't defined for the type 'ImageData'.

[Mortal-HY]

This is how CameraImage is handled in the iOS environment

Under the startImageStream function

var bytes = cameraImg.planes[0].bytes;
Map<String, dynamic> json = {};
json["bytes"] = bytes;
json['bytesPerRow'] = cameraImg.planes[0].bytesPerRow;
json['height'] = cameraImg.planes[0].height;
json['width'] = cameraImg.planes[0].width;
json['raw'] = cameraImg.format.raw as int;
Uint8List? result = await channel.invokeMethod('invokeConvert', json);

Called in the callback of iOS's MethodChannel

if([method isEqualToString:@"invokeConvert"]){
    NSData *data = [ConvertCameraImg inputDataWithJson:call.arguments];
    result(data);
}

Specific treatment methods

import Foundation
import Accelerate

struct Image: Codable {
    let width: Int
    let height: Int
    let bytesPerRow: Int
    let raw: Int
    var data: Data!
}

class ConvertCameraImg: NSObject {
    @objc public static func inputData(json: [String: Any]) -> Data? {
        var j = json
        let data = (j.removeValue(forKey: "bytes") as? FlutterStandardTypedData)!.data
        guard let jsonData = try? JSONSerialization.data(withJSONObject: j) else { return nil }
        guard var image = try? JSONDecoder().decode(Image.self, from: jsonData) else { return nil }
        image.data = data
        var buffer = [UInt8](data)
        let uint8Pointer = UnsafeMutablePointer<UInt8>.allocate(capacity: buffer.count)
        uint8Pointer.initialize(from: &buffer, count: buffer.count)
        defer {
            uint8Pointer.deallocate()
        }

        ConvertCameraImg.swizzleBGRA8toRGBA8(uint8Pointer, width: image.width, height: image.height)
        
        guard let colorSpace = CGColorSpace(name: CGColorSpace.linearSRGB) else { return nil }
        let bitmapInfo = CGImageAlphaInfo.premultipliedLast.rawValue
        guard let bitmapContext = CGContext(data: nil,
                                            width: image.width,
                                            height: image.height,
                                            bitsPerComponent: 8,
                                            bytesPerRow: image.bytesPerRow,
                                            space: colorSpace,
                                            bitmapInfo: bitmapInfo) else { return nil }
        bitmapContext.data?.copyMemory(from: buffer, byteCount: buffer.count)
        
        guard let img = bitmapContext.makeImage() else { return nil }
        
        
        guard let imageData = CFDataCreateMutable(kCFAllocatorDefault, 0) else { return nil }
        guard let dest = CGImageDestinationCreateWithData(imageData, "public.jpeg" as CFString, 1, nil) else { return nil }
        CGImageDestinationAddImage(dest, img, nil)
        let result = CGImageDestinationFinalize(dest)
        if result {
            return imageData as Data
        }
        return nil
    }
    
    static func swizzleBGRA8toRGBA8(_ bytes: UnsafeMutableRawPointer, width: Int, height: Int) {
        var sourceBuffer = vImage_Buffer(data: bytes,
                                         height: vImagePixelCount(height),
                                         width: vImagePixelCount(width),
                                         rowBytes: width * 4)
        var destBuffer = vImage_Buffer(data: bytes,
                                       height: vImagePixelCount(height),
                                       width: vImagePixelCount(width),
                                       rowBytes: width * 4)
        var swizzleMask: [UInt8] = [ 2, 1, 0, 3 ] // BGRA -> RGBA
        vImagePermuteChannels_ARGB8888(&sourceBuffer, &destBuffer, &swizzleMask, vImage_Flags(kvImageNoFlags))
    }
}

Android processing can use the YUVTransform plugin

[guyluz11]

This is a sample of what the CameraImage contains:

I/flutter (27030): Format group: ImageFormatGroup.yuv420
I/flutter (27030): height: 480
I/flutter (27030): width: 640
I/flutter (27030): Number of planes: 3
I/flutter (27030): plane 0 length: 307200
I/flutter (27030): plane 0 data: [16, 21, 21, 19, 19,
I/flutter (27030): plane 1 length: 76800
I/flutter (27030): plane 1 data: [130, 126, 126, 125,
I/flutter (27030): plane 2 length: 76800
I/flutter (27030): plane 2 data: [125, 131, 131, 132,

How can I convert this to a Uint8List acceptable to Image.memory()?? Or is there another widget which can display a CameraImage?

I am using

Codec codec = await instantiateImageCodec(ulist)
image = (await codec.getNextFrame()).image

and you can show the image like this

RawImage(image: image)
And it works for normal images.

But like others here I am getting errors in the conversion when using the images from the stream

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Exception: Invalid image data
#0      _futurize (dart:ui/painting.dart:6950:5)
#1      ImageDescriptor.encoded (dart:ui/painting.dart:6764:12)
#2      instantiateImageCodecWithSize (dart:ui/painting.dart:2307:60)
#3      instantiateImageCodecFromBuffer (dart:ui/painting.dart:2251:10)
#4      instantiateImageCodec (dart:ui/painting.dart:2198:10)
<asynchronous suspension>
#5      _ImageStreamPageState.videoStream.<anonymous closure> (package:cbj_smart_device_flutter/presentation/client/image_stream_page.dart:39:11)
<asynchronous suspension>

[GleammerRay]

@faslurrajah this solution no longer works on latest package version due to API changes. Here's an adjusted series of methods from my project, ready for general use and should work on both Android and iOS:

// CameraImage BGRA8888 -> PNG
// Color
imglib.Image imageFromBGRA8888(CameraImage image) {
  return imglib.Image.fromBytes(
    width: image.width,
    height: image.height,
    bytes: image.planes[0].bytes.buffer,
    order: imglib.ChannelOrder.bgra,
  );
}

// CameraImage YUV420_888 -> PNG -> Image (compresion:0, filter: none)
// Black
imglib.Image imageFromYUV420(CameraImage image) {
  final uvRowStride = image.planes[1].bytesPerRow;
  final uvPixelStride = image.planes[1].bytesPerPixel ?? 0;
  final img = imglib.Image(width: image.width, height: image.height);
  for (final p in img) {
    final x = p.x;
    final y = p.y;
    final uvIndex =
        uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
    final index = y * uvRowStride +
        x; // Use the row stride instead of the image width as some devices pad the image data, and in those cases the image width != bytesPerRow. Using width will give you a distored image.
    final yp = image.planes[0].bytes[index];
    final up = image.planes[1].bytes[uvIndex];
    final vp = image.planes[2].bytes[uvIndex];
    p.r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255).toInt();
    p.g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91)
        .round()
        .clamp(0, 255)
        .toInt();
    p.b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255).toInt();
  }

  return img;
}

imglib.Image? imageFromCameraImage(CameraImage image) {
  try {
    imglib.Image img;
    switch (image.format.group) {
      case ImageFormatGroup.yuv420:
        img = imageFromYUV420(image);
        break;
      case ImageFormatGroup.bgra8888:
        img = imageFromBGRA8888(image);
        break;
      default:
        return null;
    }
    return img;
  } catch (e) {
    //print(">>>>>>>>>>>> ERROR:" + e.toString());
  }
  return null;
}

[thandal]

FYI: I think I have an explanation. While some phones produce yuv420p, others actually quietly are producing nv21/nv12, but then creating interleaved U and V planes.

• Y plane -- always good, always the expected size
• if U and V planes have 2 bytesPerPixel (better named pixelStride in the Android CameraX documentation), they are actually interleaved U and V channels

This means the U and V bytes are interleaved in memory as follows, and the U and V plane buffers are just offset by one byte:

[ U1 V1 U2 V2 ....  Un Vn]  <-- actual layout in memory, with length (w/2) * (h/2) * 2
 ^-- U buffer starts, pixelStride is 2
     ^-- V buffer starts, pixelStride is 2
                     ^-- U buffer ends, size is odd
                        ^-- V buffer ends, size is odd

Which is actually nv21 (or nv12 because U is first) "faking it" as yuv420p! So everything makes sense, and you can actually get nv21 just by using the whole (interleaved) U buffer, and the last byte of the V buffer! Of course, it would be best if we could actually get "explicit" n21.

[alexcohn]

@thandal I am not sure what you want to achieve this way. Is there some API that expects an nv12 contiguous buffer?