[RFC] Java Camera2 View for Android #10081

floe · 2017-11-13T12:01:39Z

Based on #10050, here's a first alpha implementation of the Camera2 adapter class which should allow zero-copy access to preview image data. Comments very welcome.

Known issues at the moment:

tutorial2-mixedprocessing works with this class as a drop-in replacement for JavaCameraView, but quits after ~ 20-30 seconds (no crash, just exit due to internal camera queue error, perhaps some race condition?) - fixed, was a memory leak
lots of code overlap with JavaCameraView and Camera2Renderer (e.g. JavaCameraFrame internal class), needs cleanup/refactoring
cacPreviewSize(...) currently buggy, selects smaller preview size than feasible

force_builders=Android pack

alalek · 2017-11-13T12:52:13Z

modules/java/generator/src/java/android+JavaCamera2View.java

+                    if (image == null) return;
+                    ByteBuffer plane = image.getPlanes()[0].getBuffer();
+                    Mat yuv = new Mat( h * 3/2, w, CvType.CV_8UC1, plane );
+                    JavaCamera2Frame tempFrame = new JavaCamera2Frame(yuv,w,h);


This says that all three planes should be used for YUV_420_888.

BTW, Please add additional sanity checks (assertions) - for actual Image format, ByteBuffer size, etc.

Hmm... OTOH, you're correct (there are 3 planes), but obviously, the planes are contiguous in memory and also mapped in one single chunk, otherwise, the cvtColor call would cause a segfault.

To implement this in a way that references all 3 planes separately would AFAICT require a new implementation of cvtColor that can handle 3 separate Mat objects. Otherwise, you'd need to copy the planes together into a single Mat and lose the advantage of the zero-copy native buffer.

We have hal:: conversion functions accepting two separate pointers for Y and UV planes (example), so you can try to use them. You can add more overloaded functions if needed. But it is hard to say how it will work taking in account that each plane has its own PixelStride parameter which is not supported by current implementations.

According to https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888 the Y plane will always have a stride of 1, and the U/V planes will have equal stride. I may simply assert that U/V stride is also 1 for the moment (seems to be true for all implementations I've seen so far).

Are you sure U and V have PixelStride=1? Probably it is 2. If your current code worked with COLOR_YUV2RGBA_NV12 conversion by providing only Y pointer, then planes should be located in continuous memory block like this:

YYYY YYYY UVUV

Then:

Uptr = Yptr + row_stride * rows Upixel_stride = 2 Vptr = Uptr + 1 Vpixel_stride = 2

My mistake, you're right. So it's one separate Y plane, directly followed by one interleaved V/U plane.

But in that case, cvtTwoPlaneYUVtoBGR should probably work?

Probably it should, but we should check these assumptions (via CV_Assert) before calling any processing code.

@floe , the problem with cvtTwoPlaneYUVtoBGR is that it does not get wrapped automatically, because it accepts raw C++ pointers. So you will need to write JNI wrapper by hand or wait_for/propose extended cvtColor(InputArray ...) interface.

Is there any way to check whether the planes are contiguous in memory before passing them to Mat()? Then the current approach would already be sufficient...

alalek · 2017-11-13T12:54:33Z

modules/java/generator/src/java/android+JavaCamera2View.java

+        }
+    }
+
+    boolean cacPreviewSize(final int width, final int height) {


There is typo (in original code too): calcPreviewSize

Aaah. This is supposed to mean "calc". 🤦‍♂️ Will fix that.

alalek · 2017-11-13T13:00:36Z

Please update files filtering for old Android versions here

floe · 2017-12-23T14:21:41Z

I've updated this PR:

fixed memory leak
fixed some typos
fixed incorrect format handling
added lots of asserts

The only thing that remains open IMHO is the handling of the RGB conversion here: https://github.com/opencv/opencv/pull/10081/files#diff-980f99834f5223fab7fde83308fc9930R353

AFAICT the cleanest way would be to store Y and U/V plane as two Mat objects in JavaCamera2Frame and add a wrapper to ImgProc.cvtColor that calls through to cvtTwoPlaneYUVtoBGR. Opinions?

floe · 2017-12-28T16:18:03Z

Side question: where are the Java wrappers for cvtColor defined?

alalek · 2017-12-28T16:41:42Z

Almost all wrapper for Java/Python bindings are generated automatically from public headers of modules (functions with CV_EXPORTS_W).

BTW, cv::Mat doesn't support multiple planes (you need 2 cv::Mat to store Y and UV planes).

…amera2-view

floe · 2018-01-01T17:33:51Z

Another side question: I added

force_builders=Android pack

to the initial PR comment, but that still doesn't seem to trigger the Android pack build?

floe · 2018-01-15T08:50:24Z

bump sorry for the noise, but I do need the Android pack to work for testing...

…amera2-view

floe · 2018-01-15T10:06:06Z

Nevermind, seems to work now 👍

…amera2-view

floe · 2018-01-29T08:34:01Z

I think this should now be ready for merging. I still can't get the "Android pack" build to trigger reliably, though. Any comments welcome /cc @alalek @mshabunin @vpisarev

…amera2-view

alalek

Well done! Works great with Android emulator (x86)!

alalek · 2018-02-10T21:44:15Z

modules/java/generator/android-21/java/org/opencv/android/JavaCamera2View.java

+                    "Interrupted while createCameraPreviewSession", e);
+        }
+        finally {
+            //mCameraOpenCloseLock.release();


What is about this line?

What purpose of this semaphore?

It was already part of the original JavaCameraView, so I've kept it. However, it's quite possible that this is a kludge from the Camera1 API that's no longer needed for Camera2. I'll see if I can just get rid of it (I'd assume yes ;-)

Agreed.
Looks like locks are no longer needed (frames are processed in "sync" with callback with Camera2 API).

…amera2-view

floe · 2018-02-13T12:04:06Z

Removed the old semaphore and ran another test; seems to work just as well. Ready for merging IMHO.

floe · 2018-02-14T12:12:10Z

Hm. The Linux OpenCL build failed, looks like a transient build error?

alalek · 2018-02-14T12:34:05Z

Build failure is not related to your patch.

I run some tests with JavaCamera2View interface via OpenCV "tutorial2" sample and observing problem with application pause/resume events processing (pressing Android 'Home' button). Does it work for you (pause/restore handling)?

floe · 2018-02-15T08:21:07Z

I'll have to test that again. What would be the expected behaviour with the old JavaCameraView?

alalek · 2018-02-15T09:14:09Z

expected behaviour

Application activity pause/resume handling should work with app crash/closing

- re-apply Java code style - imports cleanup - dump expections information

alalek · 2018-02-15T16:10:46Z

@floe I pushed several updates to this patch. Please take a look.

alalek

@floe Thank you for the contribution!

floe · 2018-02-17T13:47:55Z

Thanks for your help!

PhilLab · 2018-03-08T16:09:42Z

@floe Awesome, can't wait to try it. Do you know if the current implementation respects the camera orientation? #9269 and #8866

floe · 2018-03-24T16:33:22Z

@PhilLab sorry for the late reply, have in fact not tried this yet. But since the code is based on CameraBridgeViewBase too, I suspect it will have the same issues.

ValkA · 2019-01-04T23:01:20Z

I have an Android device that only works with ImageReader of ImageFormat.YV12.
The output is 3 planes as following:

Mat y_mat = new Mat(h, w, CvType.CV_8UC1, image.getPlanes()[0].getBuffer());
Mat u_mat = new Mat(h/2, w/2, CvType.CV_8UC1, image.getPlanes()[1].getBuffer());
Mat v_mat = new Mat(h/2, w/2, CvType.CV_8UC1, image.getPlanes()[2].getBuffer());

Do you have an idea how to convert all these into a full sized RGBA without copying/merging matrices?

ValkA · 2019-01-04T23:28:18Z

Going with @floe assumption that native image buffer is contiguous I'm getting desired results with:

Mat yuv420 = new Mat(h+h/2, w, CvType.CV_8UC1, image.getPlanes()[0].getBuffer());
Imgproc.cvtColor(yuv420, rgba, Imgproc.COLOR_YUV2RGBA_YV12, 4);

ValkA · 2019-01-04T23:44:01Z

Ok inspired by JavaCameraView, this one listener handles both image formats :)
Emulator also.

ImageReader.OnImageAvailableListener listener = new ImageReader.OnImageAvailableListener {
        final Mat rgba;
        int w,h;

        public OnImageAvailableListener(int w, int h){
            this.w = w;
            this.h = h;
            this.rgba = new Mat(h,w,CvType.CV_8UC4);
        }

        @Override
        public void onImageAvailable(ImageReader imageReader) {
            Image image = imageReader.acquireLatestImage();
            if(image == null) return;

            Mat yuv420 = new Mat(h+h/2, w, CvType.CV_8UC1, image.getPlanes()[0].getBuffer());
            switch (imageReader.getImageFormat()){
                case ImageFormat.YV12:
                    Imgproc.cvtColor(yuv420, rgba, Imgproc.COLOR_YUV2RGBA_YV12, 4);
                    break;
                case ImageFormat.YUV_420_888:
                    Imgproc.cvtColor(yuv420, rgba, Imgproc.COLOR_YUV2BGRA_NV12, 4);
                    break;
            }

            //Do your thing here...

            Utils.matToBitmap(rgba, bitmap);
            image.close();
            cameraView.setImageBitmap(bitmap);
        }
    }

ansman · 2019-02-20T20:06:27Z

@ValkA Your code only uses the Y plane which would lead to a gray scale picture.

ValkA · 2019-02-20T20:29:31Z

It takes also UV in the extra h/2 rows

ansman · 2019-02-20T20:42:23Z

Ah, I see. It does seem to be continuous but that is probably a dangerous assumption that can lead to strange issues down the line.

ValkA · 2019-02-20T20:45:45Z

Yeah, also when lines are padded it has issues

ansman · 2019-05-07T22:21:16Z

@ValkA, @floe, @alalek I've done some experiments and it seems that if you use a different size for your image reader than your surface (or perhaps it's just different from the standard 1920x1080) you don't actually get NV21 because of padding between the rows.

Furthermore it seems that V always comes before U in memory so just taking the second plane as an uv buffer seems incorrect (if anything it should take the third plane and treat it as vu)

alalek · 2019-05-07T22:50:43Z

@ansman Could you please check this patch on your device: #14505 ?

ansman · 2019-05-09T11:14:12Z

@alalek I added a review for the PR

floe added 4 commits November 13, 2017 11:26

add (untested) JavaCamera2View class

dfe3a77

initial fixes

4c3ef6e

minor cleanup

0aa8b31

Merge branch 'master' into java-camera2-view

e727101

alalek reviewed Nov 13, 2017

View reviewed changes

floe added 7 commits November 13, 2017 15:11

exclude JavaCamera2View from build for older SDK version

20f0577

Merge branch 'master' into java-camera2-view

9130b5a

Merge remote-tracking branch 'upstream/master' into java-camera2-view

558794b

fix method name typo

5e291b2

add asserts + sanity checks

4cb65a9

fix preview format checks

58c75b8

fix the memory leak

db27b40

floe added 3 commits December 30, 2017 15:32

export cvtTwoPlaneYUVtoBGR for Java usage

c5cc3f3

Merge branch 'master' of https://github.com/opencv/opencv into java-c…

c3cb2c4

…amera2-view

add handling for two-plane YUV frames (C wrapper still missing)

b285933

Merge branch 'master' of https://github.com/opencv/opencv into java-c…

3e28074

…amera2-view

floe added 4 commits January 24, 2018 21:44

add two-plane cvtColor helper function

6e7d3da

Merge branch 'master' of https://github.com/opencv/opencv into java-c…

78777d9

…amera2-view

fix warnings

11366ca

actually use the new cvtColorTwoPlane helper func

5f86735

floe added 2 commits February 2, 2018 15:26

Merge branch 'master' of https://github.com/opencv/opencv into java-c…

16c6fbc

…amera2-view

fix wrong output matrix size

1d378c6

alalek reviewed Feb 10, 2018

View reviewed changes

floe added 2 commits February 13, 2018 12:59

Merge branch 'master' of https://github.com/opencv/opencv into java-c…

33d2086

…amera2-view

remove leftover semaphore from camera api 1

d34c099

alalek and others added 3 commits February 15, 2018 18:50

android: JavaCamera2View minor refactoring

8354583

- re-apply Java code style - imports cleanup - dump expections information

android: JavaCamera2View: fix mScale

b07ded8

android: JavaCamera2View: pause/resume fixes

8f06520

alalek mentioned this pull request Feb 15, 2018

Android Camera2 API improvements #10882

Closed

alalek approved these changes Feb 17, 2018

View reviewed changes

alalek merged commit 6a8f57e into opencv:master Feb 17, 2018

mshabunin mentioned this pull request May 10, 2018

cvtColorTwoPlane #11492

Closed

alalek mentioned this pull request May 7, 2019

android: fix JavaCamera2View UV plane handling #14505

Merged

[RFC] Java Camera2 View for Android #10081

[RFC] Java Camera2 View for Android #10081

Conversation

floe commented Nov 13, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mshabunin Nov 13, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mshabunin Nov 13, 2017 • edited

Choose a reason for hiding this comment

floe Nov 14, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alalek commented Nov 13, 2017

floe commented Dec 23, 2017

floe commented Dec 28, 2017

alalek commented Dec 28, 2017

floe commented Jan 1, 2018

floe commented Jan 15, 2018

floe commented Jan 15, 2018

floe commented Jan 29, 2018

alalek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

floe commented Feb 13, 2018

floe commented Feb 14, 2018

alalek commented Feb 14, 2018

floe commented Feb 15, 2018

alalek commented Feb 15, 2018

alalek commented Feb 15, 2018

alalek left a comment

Choose a reason for hiding this comment

floe commented Feb 17, 2018

PhilLab commented Mar 8, 2018

floe commented Mar 24, 2018

ValkA commented Jan 4, 2019

ValkA commented Jan 4, 2019 • edited

ValkA commented Jan 4, 2019 • edited

ansman commented Feb 20, 2019

ValkA commented Feb 20, 2019

ansman commented Feb 20, 2019

ValkA commented Feb 20, 2019

ansman commented May 7, 2019

alalek commented May 7, 2019

ansman commented May 9, 2019

floe commented Nov 13, 2017 •

edited

mshabunin Nov 13, 2017 •

edited

mshabunin Nov 13, 2017 •

edited

floe Nov 14, 2017 •

edited

ValkA commented Jan 4, 2019 •

edited

ValkA commented Jan 4, 2019 •

edited