[camera] Access realtime image frames for pre-processing and effect overlays

[rubencg195]

Is it possible to access the real-time camera frame feed to do pre-processing or to be used for ML implementations like face-recognition, Image overlays, filters, etc.?

[zoechi]

Sounds like https://github.com/flutter/plugins/tree/master/packages/camera can do that but I haven't tried myself yet.

[rubencg195]

Thanks, but it will be better if you can obtain the byte array to work with instead of writing in memory and then retrieving the image for preprocessing. It can be costly in time and resources. I think it can be a really good feature to implement. Just think of the results, ML application, filters, AR/VR application, etc.

Maybe includying a Opencv wrapper for flutter. If possible, it could be awesome!
https://docs.opencv.org/2.4/doc/tutorials/introduction/android_binary_package/O4A_SDK.html

[dessant]

Have you found a workaround for this? I'm looking for a way to grab the camera frames to then process them with ML Kit.

[Robbo2]

I'm also looking for a way to access real-time camera frame feed to do pre-processing and live-streaming.
Anyone have suggestions?

[zoechi]

There is some work going on that looks related flutter/plugins#679

[dustin-graham]

Thanks @zoechi, I'm working to get that PR into shape soon. It'll be a good solution for working with ML Kit.

[arloan]

I need this also. The camera plugin can only take a camera shot into image file, which is really too much time consuming for many kinds of scenario.

[dustin-graham]

@arloan I made a prototype with sending live frames from the camera plugin to another plugin. Would you mind taking a look, offering suggestions and feedback? flutter/plugins#688

…

On Mon, Jul 30, 2018, 6:43 PM arloan ***@***.***> wrote: I need this also. The camera plugin can only take a camera shot into image file, which is really too much time consuming for many kinds of scenario. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18968 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA6lu5FXIXsyYRL4r6XsJyJ5HjPCFTA2ks5uL7ZdgaJpZM4U-DJr> .

[arloan]

Well, I'm afraid I don't have enough knowledge of Java or Android, that's why I chose flutter ...

[Nashua354]

Maurice added the bytestream capability in the camera package
flutter/plugins#965
and modified the firebase_ml_vision package too to make this demo
https://github.com/bparrishMines/mlkit_demo
Check it out. Its some amazing stuff.

[alejamp]

Camera plugin adds startImageStream in CameraController sice 0.2.9. The image format returned is YUV420-888 which is pretty useful for ML Kit, but not to show on screen.
I wrote a piece of code to convert the plane[0] from YUV420 to PNG using the DART Image library in order to use Image Widget to display it... but it takes too long almost 800ms per frame.

[gimox]

@alejamp i'm working on custom model with ml_vision. Can you share the code to convert image?

[alejamp]

@alejamp i'm working on custom model with ml_vision. Can you share the code to convert image?

Here is the code

#26348 (comment)

[gimox]

@alejamp thanks for code. now i have a ios custom model working with firebase_ml_vision but it's too slow for a production app. I think the only solution is working with native image manipulation.

any suggestions is appreciated about ios and android code....

[rubencg195]

Hello everyone, is there any production-ready version (IOS & Android) for this feature?

[gimox]

not for now!

[rubencg195]

Thank you @gimox. How is ML-kit able to recognize faces and text in near real-time? Is there a way to access the image-stream from that package?

[gisinator]

Plus 1 for this. Trying to take 20 pictures without having to save them first on disk, so access to frames in a useful way would be super exciting.

[jdeltoft]

Was this resolved ever?

[Baileypollard]

Any updates on a solution for this?

[rubencg195]

@zoechi any update? thank you.

[brainstorm]

Just googling a bit: https://github.com/shaqian/flutter_realtime_detection

... would that help?

[SelanDeemantha]

Hei, is there a proper way to process images(YUV420 (YCbCr)) which are captured by live-streaming to use for object recognition ?

I have used https://pub.dev/documentation/google_ml_kit/latest/ for image detection, and I need to crop images from frames in camera live stream to process for image recognition. I have referred below docs https://medium.com/@hugand/capture-photos-from-camera-using-image-stream-with-flutter-e9af94bc2bee
https://medium.com/@vad.pinchuk/image-stream-processing-in-flutter-application-by-tflite-neural-networks-2c40d65f3b67
If someone knows a better solution to achieve this using dart, please give some advice to follow up.

[danielalonsomendez]

Any update on this?