Add byte streaming capability for the camera #965

bparrishMines · 2018-12-06T22:02:17Z

No description provided.

…_android_2

…id_2

bparrishMines · 2018-12-06T22:17:47Z

packages/camera/android/src/main/java/io/flutter/plugins/camera/CameraPlugin.java

@@ -258,7 +278,8 @@ public boolean onRequestPermissionsResult(int id, String[] permissions, int[] gr
    private CameraDevice cameraDevice;
    private CameraCaptureSession cameraCaptureSession;
    private EventChannel.EventSink eventSink;
-    private ImageReader imageReader;
+    private ImageReader pictureImageReader;
+    private ImageReader byteImageReader; // Used to pass bytes to dart side.


We need to separate readers because they use different formats.

This should be explained in a comment in the code.

bparrishMines · 2018-12-06T22:21:20Z

packages/camera/ios/Classes/CameraPlugin.m

 - (void)captureToFile:(NSString *)filename result:(FlutterResult)result;
 @end

 @implementation FLTCam
+FourCharCode const videoFormat = kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange;


Format that I believe is recommended for iOS 10+, which is minimum requirement for this plugin.

Add this explanation as a comment for future readers to understand the code.

bparrishMines · 2018-12-06T22:25:13Z

packages/camera/ios/Classes/CameraPlugin.m

@@ -269,7 +337,66 @@ - (CVPixelBufferRef)copyPixelBuffer {
  while (!OSAtomicCompareAndSwapPtrBarrier(pixelBuffer, nil, (void **)&_latestPixelBuffer)) {
    pixelBuffer = _latestPixelBuffer;
  }
-  return pixelBuffer;
+
+  return [self convertYUVImageTOBGRA:pixelBuffer];


Since video format was changed to kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange we have to convert image to a usable format for flutter textures. Which is kCVPixelFormatType_32BGRA.

dwach414 · 2018-12-14T01:21:18Z

@bparrishMines
Ya I think the iPhone Xs thing is either a bug with the CameraPreview specifically or something unrelated that I caused. The OCR business logic still works on preset high even when the image doesn't show up. It was still broken as well when I reverted to 0.2.6. I'll do some more digging on that end.

sigurdm

LGTM.

I would like for some of the other current plugin-maintainers to also review this.

I worry about the efficiency of transferring the data across the messageChannel.
I guess we do not have a lot of choice currently - but it seems very heavy, and not the intended use.

A future development would be to have a platform-image plugin (or image-stream plugin) that allows the images to stay on the native side, and be referenced from dart. It could then provide a number of compositions of images, and the ability to project an image into a Texture, or - if really needed - transfer the bytes to dart.

The camera-plugin would know about the image-plugin and be able to send the images there...

sigurdm · 2018-12-14T09:55:54Z

packages/camera/android/src/main/java/io/flutter/plugins/camera/CameraPlugin.java

@@ -258,7 +278,8 @@ public boolean onRequestPermissionsResult(int id, String[] permissions, int[] gr
    private CameraDevice cameraDevice;
    private CameraCaptureSession cameraCaptureSession;
    private EventChannel.EventSink eventSink;
-    private ImageReader imageReader;
+    private ImageReader pictureImageReader;
+    private ImageReader byteImageReader; // Used to pass bytes to dart side.


This should be explained in a comment in the code.

sigurdm · 2018-12-14T09:56:51Z

packages/camera/example/ios/Runner.xcodeproj/project.pbxproj

@@ -183,7 +182,7 @@
 				TargetAttributes = {
 					97C146ED1CF9000F007C117D = {
 						CreatedOnToolsVersion = 7.3.1;
-						DevelopmentTeam = EQHXZ8M8AV;
+						DevelopmentTeam = S8QB4VV633;


I don't think changes to this file should be checked in.

sigurdm · 2018-12-14T09:57:44Z

packages/camera/ios/Classes/CameraPlugin.m

 - (void)captureToFile:(NSString *)filename result:(FlutterResult)result;
 @end

 @implementation FLTCam
+FourCharCode const videoFormat = kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange;


Add this explanation as a comment for future readers to understand the code.

packages/camera/lib/camera.dart

sigurdm · 2018-12-14T10:43:14Z

packages/camera/lib/src/camera_image.dart

+///
+/// Although not all image formats are planar on iOS, we treat 1-dimensional
+/// images as single planar images.
+class CameraImage {


Perhaps add a helper to make it easy to turn this into a https://docs.flutter.io/flutter/painting/ImageProvider-class.html

The current implementation doesn't include a compatible image format with 'ImageProvider'. This impl uses yuv420.

I also include in CameraImage comments that The [CameraImage] is not directly usable as a UI resource.

packages/camera/CHANGELOG.md

packages/camera/android/src/main/java/io/flutter/plugins/camera/CameraPlugin.java

sigurdm · 2018-12-14T10:50:38Z

packages/camera/lib/src/camera.dart

+  ///
+  /// Throws a [CameraException] if byte streaming or video recording has
+  /// already started.
+  Future<void> startByteStream(OnLatestImageAvailable onAvailable) async {


Should this also take a resolution, and a fps-setting?
It seems many times you would want a lower resolution setting for the bytestream

Due to a limitation I ran into on iOS. I wouldn't be able to have a different resolution for image streaming and the [CameraPreview] widget.

Although, we could scale the image on the platform side before we send it over the platform channel. I think this additional code would be best in a separate PR though.

FPS Setting could be added as well. I added a TODO to add these options.

packages/camera/lib/src/camera.dart

mikuhl-dev · 2018-12-15T16:32:03Z

Need this for barcode scanning!

slightfoot · 2018-12-18T12:23:46Z

@bparrishMines Do you know when this is going to be merged?

mikuhl-dev · 2018-12-21T06:45:58Z

@bparrishMines I am getting java.lang.IllegalStateException: Session has been closed; further changes are illegal. whenever I open the camera the second time. If I DON'T dispose the controller, this does not happen.

lifenautjoe · 2018-12-27T06:07:56Z

Need this for mustache filters !

…id_2

bparrishMines · 2019-01-03T01:42:31Z

@MichaelPriebe I wasn't running into any issues for with the example camera app. What do you mean by 'open the camera the second time'. Are you creating a new CameraController and calling initialize or are you creating a new CameraPreview. I would also check that you are using await with the dispose() method and initialize() method.

Alby-o · 2019-01-17T00:54:47Z

@bparrishMines Thanks for this! I was wondering if there's a way to convert the CameraImage for an API I wanted to use. The requirements:

Blob of image bytes up to 5 MBs.
Type: Base64-encoded binary data object

Alternatively if I could save the image to a file that would also be great! Thanks

This reverts commit b40042f.

ollyde · 2020-04-29T13:28:48Z

Hey guys, the image stream is not sufficient for many use cases and doesn't include audio sources by default (like live RMTP/HLS streams). Is there any possibility to get startByteStream? At the moment we save to file but as soon as something like FFMPEG accesses that file the camera stops streaming too it :-(

PanikPlunder · 2020-07-12T18:10:31Z

@bparrishMines Thanks for this! I was wondering if there's a way to convert the CameraImage for an API I wanted to use. The requirements:

Blob of image bytes up to 5 MBs.

Type: Base64-encoded binary data object

Alternatively if I could save the image to a file that would also be great! Thanks

You could encode the camera image to an image like jpg and send the bytes as blob or encode it to base64 and send it
Example:

Repo: https://pub.dev/packages/image

Example code:
CameraImage image
imglib.Image img = imglib.Image.fromBytes(image.planes[0].width, image.planes[0].height, image.planes[0].bytes);

bparrishMines added 17 commits October 24, 2018 12:56

Start of Android side of byte stream passing

c3f703e

dart side of byte streaming

f310580

Fix android streaming

87fdb60

Merge branch 'master' of github.com:flutter/plugins into camera_android

673b003

Add ios byte streaming

8d353ad

Convert buffer to uiimage to pass over

0291a29

formatting

31f746a

Stream yuv bytes instead

5fcfeb2

Make video format a constant

040d1ae

Pass back metadata for ios image

16d1d32

Pass back metadata for android image

8e88978

Dart code now parses camera image buffer

67f8304

YUV image to bgra

646283a

Add documentation

7e99691

Only pass available data on Android

4cbfab9

Merge branch 'master' of github.com:bparrishMines/plugins into camera…

aa0e263

…_android_2

Merge branch 'master' of github.com:flutter/plugins into camera_andro…

9b2ae22

…id_2

googlebot added the cla: yes label Dec 6, 2018

bparrishMines mentioned this pull request Dec 6, 2018

Add byte streaming capability for the camera #900

Closed

Bump version

297fe7a

bparrishMines commented Dec 6, 2018

View reviewed changes

bparrishMines force-pushed the camera_android_2 branch from de28941 to 4cbb8b2 Compare December 6, 2018 22:22

bparrishMines requested a review from collinjackson December 6, 2018 22:22

bparrishMines commented Dec 6, 2018

View reviewed changes

bparrishMines force-pushed the camera_android_2 branch from 4cbb8b2 to d0f173b Compare December 7, 2018 10:12

Formatting

bdd9007

bparrishMines force-pushed the camera_android_2 branch from d0f173b to bdd9007 Compare December 7, 2018 10:22

bparrishMines mentioned this pull request Dec 7, 2018

WIP: Live MLKit vision detection using the camera plugin #688

Closed

create imageformat error

ccc057b

sigurdm approved these changes Dec 14, 2018

View reviewed changes

bparrishMines mentioned this pull request Dec 19, 2018

Camera: Please allow access to images without forcing them to be saved first flutter/flutter#25087

Closed

Nashua354 mentioned this pull request Dec 19, 2018

[camera] Access realtime image frames for pre-processing and effect overlays flutter/flutter#18968

Open

bparrishMines added 9 commits December 19, 2018 14:09

Better name and comments

045dd53

Change from library camera file

01520fe

bytestream -> imagestream

6155620

Comments and names

5898b4c

Formatting

7617bb9

Added resolution and fps todo

f18db98

Unmodify file

0bf466a

Empty commit to rerun tests

a53222b

Remove TODO from documentation

fe965d1

Merge branch 'master' of github.com:flutter/plugins into camera_andro…

7c0228f

…id_2

bparrishMines merged commit efe4c1b into flutter:master Jan 3, 2019

bparrishMines deleted the camera_android_2 branch January 3, 2019 20:55

mklim mentioned this pull request Jan 15, 2019

camera lib 0.2.8: CameraPreview shows black screen instead of camera preview on some iOS devices, including iPhone X flutter/flutter#26567

Closed

andreidiaconu pushed a commit to andreidiaconu/plugins that referenced this pull request Feb 17, 2019

Add byte streaming capability for the camera (flutter#965)

b40042f

andreidiaconu added a commit to andreidiaconu/plugins that referenced this pull request Feb 17, 2019

Revert "Add byte streaming capability for the camera (flutter#965)"

a4f73c1

This reverts commit b40042f.

truongsinh mentioned this pull request Feb 26, 2019

Documentation / API for camera flutter/flutter#28515

Closed

marslord mentioned this pull request Apr 15, 2019

Live Text Recognition flutter/flutter#29559

Closed

Akachu pushed a commit to Akachu/flutter_camera that referenced this pull request Apr 27, 2020

Add byte streaming capability for the camera (flutter#965)

3bf75b3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add byte streaming capability for the camera #965

Add byte streaming capability for the camera #965

bparrishMines commented Dec 6, 2018

bparrishMines Dec 6, 2018

sigurdm Dec 14, 2018

bparrishMines Dec 20, 2018

bparrishMines Dec 6, 2018

sigurdm Dec 14, 2018

bparrishMines Dec 20, 2018

bparrishMines Dec 6, 2018

dwach414 commented Dec 14, 2018 •

edited

Loading

sigurdm left a comment

sigurdm Dec 14, 2018

sigurdm Dec 14, 2018

sigurdm Dec 14, 2018

sigurdm Dec 14, 2018

bparrishMines Dec 20, 2018

sigurdm Dec 14, 2018

bparrishMines Dec 20, 2018

bparrishMines Dec 20, 2018

mikuhl-dev commented Dec 15, 2018

slightfoot commented Dec 18, 2018

mikuhl-dev commented Dec 21, 2018 •

edited

Loading

lifenautjoe commented Dec 27, 2018

bparrishMines commented Jan 3, 2019

Alby-o commented Jan 17, 2019

ollyde commented Apr 29, 2020

PanikPlunder commented Jul 12, 2020

Add byte streaming capability for the camera #965

Add byte streaming capability for the camera #965

Conversation

bparrishMines commented Dec 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dwach414 commented Dec 14, 2018 • edited Loading

sigurdm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikuhl-dev commented Dec 15, 2018

slightfoot commented Dec 18, 2018

mikuhl-dev commented Dec 21, 2018 • edited Loading

lifenautjoe commented Dec 27, 2018

bparrishMines commented Jan 3, 2019

Alby-o commented Jan 17, 2019

ollyde commented Apr 29, 2020

PanikPlunder commented Jul 12, 2020

dwach414 commented Dec 14, 2018 •

edited

Loading

mikuhl-dev commented Dec 21, 2018 •

edited

Loading