Android NDK camera support #19597

komakai · 2021-02-22T11:41:47Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with OpenCV
The PR is proposed to proper branch
There is reference to original bug report and related work

This PR fixes #11952

Add an Android native camera capture backend to the videoio module.
Based loosely on https://github.com/android/ndk-samples/tree/main/camera
Requires Android native API level 24 or above (in order to access NDK camera APIs)

android_pack_config=ndk-18-api-level-24.config.py

…e callback and retry

alalek

Well done! Thank you for contribution 👍

jumpy88 · 2021-02-26T12:19:10Z

So, should now with this update cv::VideoCapture.open(0) succeed and return an handle to accede a camera on Android? Does it only work for internal cameras or also for USB cameras? (Camera2 API I understand this patch is based on should support USB cameras https://source.android.com/devices/camera/external-usb-cameras)

komakai · 2021-02-27T02:50:38Z

@jumpy88 it will work for any device that is accessible to the Android NDK Camera API (https://developer.android.com/ndk/reference/group/camera)

(Camera2 API I understand this patch is based on should support USB cameras https://source.android.com/devices/camera/external-usb-cameras)

This link is aimed at device manufacturers who are building the Android OS itself, not general Android app developers. (Also note: Camera2 is a Java API whereas the OpenCV videoio module is written in C++)

jumpy88 · 2021-03-05T11:34:23Z

@komakai Ops, sorry, I thought Camera2 was an API for developers and it was accessible in Android NDK. My knowledge about Android is still very poor. Ok, I'll give it a try with your code and will post the result.
Thank you for your contribution and support.

jumpy88 · 2021-03-25T13:09:33Z

Hi, I've just done some experiments with 2e2df44 commit in a Godot Engine proof of concept. It works with built-in cameras but doesn't seem to recognize external USB cameras. The only weird thing is sometimes the visualized frame appears having blue and red channels swapped, but I don't have enough information to open an issue (I don't know if it's due to OpenCV itself or is my fault). You can find the sample code here https://github.com/jumpy88/godot_ocv-camera-reader
Thank you for the great job @komakai

alalek · 2021-03-25T17:08:40Z

modules/videoio/src/cap_android_camera.cpp

+                }
+                return true;
+            case CV_CAP_PROP_CONVERT_RGB:
+                convertToRgb = (value != 0);


cv::cvtColor(yuv, out, convertToRgb ? cv::COLOR_YUV2RGB_YV12 : cv::COLOR_YUV2BGR_YV12);

BTW, meaning of this property is different (name misleads unfortunately, it is not BGR / RGB switch):

CV_CAP_PROP_CONVERT_RGB = 1 (default): VideoCapture returns frames as "BGR" 8UC3 cv::Mat.

CV_CAP_PROP_CONVERT_RGB = 0: VideoCapture returns frames in other formats, like NV12, Y8, etc (format is retrieved and/or sometimes specified through CAP_PROP_FOURCC)

@alalek OK thanks for the explanation - will prepare a fix for this when I get a minute.

Is there a space in the api to allow for the BGR/RGB switch in addition to the correct convertToRGB behaviour?

cheers, Jeff

Ay-De · 2022-02-09T21:16:59Z

Hello,

I am using opencv 4.5.4. python reports on android: after calling

cam = cv2.VideoCapture(0)
cam.isOpened(): True.

but i can't get a frame from the video stream. If i call

retval, frame = cam.read()

retval is returned as False and Frame is None. Do i miss something? The code works on windows.

Ay-De · 2022-02-11T14:37:30Z

@komakai Do you know what i am missing here?

komakai · 2022-02-12T00:00:07Z

@Ay-De running OpenCv Python on Android is not something I have tested. I would suggest to look in the Logcat output to see if there are any error messages there

Ay-De · 2022-02-12T11:02:42Z

@Ay-De running OpenCv Python on Android is not something I have tested. I would suggest to look in the Logcat output to see if there are any error messages there

Hello, i am using kivy to run python code on android. OpenCV is being imported, videocapture(0) (or 1) works also. As i said, the isOpened() function also returns true, but if i try to read a frame, with read(), that returns nothing. No error is thrown according to logcat, but the variable for frame says it is None.

komakai · 2022-02-13T10:49:11Z

@Ay-De Do you have an APK file you can share?

Ay-De · 2022-02-13T15:54:39Z

@Ay-De Do you have an APK file you can share?

Yes, here:
https://github.com/Ay-De/Kivy_Python_Camera_Demo/releases/tag/0.0.17

There is also the source code (See the main.py in develop branch. sorry it is still quite messy). The APK is compiled with debug mode enabled. If you execute the main.py on a windows computer with a webcam, it will work without errors (i tested it on a Surface Pro 7).

komakai · 2022-02-14T12:35:04Z

OK - so I tried your APK and saw lots of output like this:

2022-02-14 21:30:53.148 26028-26132/org.ayde.kivycamera V/NativeCamera: session 0x7bce246ba0 capture completed
2022-02-14 21:30:53.149 26028-26066/org.ayde.kivycamera E/NativeCamera: Unsupported format

Ay-De · 2022-02-14T20:13:35Z

Ok sorry i somehow missed that part. I think the problem lies in returned image format from the sensor? Is there a way to specify it in OpenCV? OpenCV has a flag CAP_PROP_FORMAT to fetch undecoded RAW video streams, but unfortunatly, the android camera seems to support different ones and i can't currently find a way to set it on the Android side. Would it maybe be possible to set the camera format after opening the image stream by making direct API calls?
https://developer.android.com/reference/android/graphics/ImageFormat

I am sorry if i am derailing this thread here!

komakai · 2022-02-15T11:16:40Z

@Ay-De The error message is coming from this line:
https://github.com/opencv/opencv/blob/4.x/modules/videoio/src/cap_android_camera.cpp#L324

There were some bug fixes recently in that area of the code:
68667d6
fa222b0
21d0f40
So my guess would be the build you have is missing one or more of these fixes.

Ay-De · 2022-02-15T21:16:16Z

So, first of all, thanks for your help. I really appreciate it.

I updated my files to include opencv in version 4.5.5, which include the above mentioned fixes, but it still doesn't work. May i ask you, which tools you are using to get the above mentioned logs?

2022-02-14 21:30:53.148 26028-26132/org.ayde.kivycamera V/NativeCamera: session 0x7bce246ba0 capture completed
2022-02-14 21:30:53.149 26028-26066/org.ayde.kivycamera E/NativeCamera: Unsupported format

I am using logcat for debugging but i am not seeing this errors, i guess i am maybe filtering out to much? I would like to understand the problems myself better instead of just asking and for that i think i need a better insight.

I am currently only seeing these logs:

02-15 16:48:55.923 10556 10748 I python : Run user 02-15 16:48:55.966 10556 10748 I python : [WARNING] [Config 02-15 16:48:55.966 10556 10748 I python : [WARNING] [Config 02-15 16:48:55.970 10556 10748 I python : [INFO ] [Logger 02-15 16:48:55.970 10556 10748 I python : [INFO ] [Kivy 02-15 16:48:55.970 10556 10748 I python : [INFO ] [Kivy 02-15 16:48:55.970 10556 10748 I python : [INFO ] [Python 02-15 16:48:55.970 10556 10748 I python : [Clang 02-15 16:48:55.970 10556 10748 I python : [INFO ] [Python 02-15 16:48:56.256 10556 10748 I python : [INFO ] [Factory 02-15 16:48:56.341 10556 10748 I python : [INFO ] [Image 02-15 16:48:56.517 10556 10748 I python : OpenCV 02-15 16:48:56.597 10556 10748 I python : [INFO ] [Window 02-15 16:48:56.611 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.611 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.612 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.612 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.612 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.613 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.613 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.613 10556 10748 I python : [INFO ] [GL 02-15 16:48:56.631 10556 10748 I python : [INFO ] [Window 02-15 16:48:56.632 10556 10748 I python : [INFO ] [Window 02-15 16:48:56.633 10556 10748 I python : [INFO ] [GL 02-15 16:48:57.073 10556 10748 I python : Camera 02-15 16:48:57.478 10556 10748 I python : Frame 02-15 16:48:57.478 10556 10748 I python : Frame 02-15 16:48:57.704 10556 10748 I python : [INFO ] [Text 02-15 16:48:57.746 10556 10748 I python : [WARNING] [Base 02-15 16:48:57.746 10556 10748 I python : [INFO ] [Base program, change dir and execute entrypoint
] Older configuration version detected (0 instead of 21)
] Upgrading configuration in progress.
] Record log in /data/user/0/org.ayde.kivycamera/files/app/.kivy/logs/kivy_22-02-15_0.txt
] v2.0.0
] Installed at "/data/user/0/org.ayde.kivycamera/files/app/_python_bundle/site-packages/kivy/init.pyc"
] v3.8.9 (default, Feb 15 2022, 16:29:31)
9.0.9 (https://android.googlesource.com/toolchain/llvm-project a2a1e703c
] Interpreter at ""
] 186 symbols loaded
] Providers: img_tex, img_dds, img_sdl2 (img_pil, img_ffpyplayer ignored)
version: 4.5.5
] Provider: sdl2
] Using the "OpenGL ES 2" graphics system
] Backend used
] OpenGL version <b'OpenGL ES 3.2 V@0530.0 (GIT@5a9022f91f, Ib11adbd47c, 1627309424) (Date:07/26/21)'>
] OpenGL vendor <b'Qualcomm'>
] OpenGL renderer <b'Adreno (TM) 660'>
] OpenGL parsed version: 3, 2
] Texture max size <16384>
] Texture max units <16>
] auto add sdl2 input provider
] virtual keyboard not allowed, single mode, not docked
] NPOT texture support is available
connected Status: True
returned: False
data: None
] Provider: sdl2
] Unknown provider
] Start application main loop

Sorry for all these probably easy to fix questions.^^

komakai · 2022-02-16T13:55:18Z

May i ask you, which tools you are using to get the above mentioned logs?
In Android Studio look for a tab along the bottom called "Logcat"

There is an Android/Java sample here:
https://github.com/komakai/NdkCamtest/blob/master/app/src/main/java/net/telepathix/ndkcam/MainActivity.java#L97
If you copy the same logic into Python it should work 🤞

Ay-De · 2022-02-20T22:26:07Z

I finally got it to work. Thanks @komakai.

asadasivam · 2023-03-08T09:56:31Z

Hi, I've just done some experiments with 2e2df44 commit in a Godot Engine proof of concept. It works with built-in cameras but doesn't seem to recognize external USB cameras. The only weird thing is sometimes the visualized frame appears having blue and red channels swapped, but I don't have enough information to open an issue (I don't know if it's due to OpenCV itself or is my fault). You can find the sample code here https://github.com/jumpy88/godot_ocv-camera-reader Thank you for the great job @komakai

Let us know any further improvements committed for external USB camera access? For external camera to support we need to enable below feature. My understanding is for some android mobile manufactures, USB Webcam access support is available, in that mobile CAMERA2 API will work. ie,. external camera will be detected in CameraManager API.

Android NDK camera support * Add native camera video backend for Android * In the event of a "No buffer available error" wait for the appropriate callback and retry * Fix stale context when creating a new AndroidCameraCapture * Add property handling

komakai added 4 commits February 20, 2021 11:29

Add native camera video backend for Android

e2022d1

In the event of a "No buffer available error" wait for the appropriat…

18317a1

…e callback and retry

Fix stale context when creating a new AndroidCameraCapture

d1cf432

Add property handling

2e2df44

alalek approved these changes Feb 25, 2021

View reviewed changes

alalek merged commit 0553543 into opencv:master Feb 25, 2021

alalek reviewed Mar 25, 2021

View reviewed changes

komakai mentioned this pull request Apr 4, 2021

Fix Android NDK camera's color format conversions #19855

Merged

3 tasks

alalek mentioned this pull request Apr 9, 2021

(5.x) Merge 4.x #19885

Merged

gabrielnhn mentioned this pull request Mar 2, 2022

Help Ay-De/Kivy_Python_Camera_Demo#7

Closed

mhsmith mentioned this pull request Sep 16, 2023

Update opencv version and add camera support chaquo/chaquopy#620

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android NDK camera support #19597

Android NDK camera support #19597

komakai commented Feb 22, 2021 •

edited by alalek

Loading

alalek left a comment

jumpy88 commented Feb 26, 2021

komakai commented Feb 27, 2021

jumpy88 commented Mar 5, 2021

jumpy88 commented Mar 25, 2021 •

edited

Loading

alalek Mar 25, 2021

komakai Mar 26, 2021

splintersilk Mar 26, 2021

Ay-De commented Feb 9, 2022

Ay-De commented Feb 11, 2022

komakai commented Feb 12, 2022

Ay-De commented Feb 12, 2022

komakai commented Feb 13, 2022

Ay-De commented Feb 13, 2022 •

edited

Loading

komakai commented Feb 14, 2022

Ay-De commented Feb 14, 2022

komakai commented Feb 15, 2022

Ay-De commented Feb 15, 2022 •

edited

Loading

komakai commented Feb 16, 2022

Ay-De commented Feb 20, 2022

asadasivam commented Mar 8, 2023

Android NDK camera support #19597

Android NDK camera support #19597

Conversation

komakai commented Feb 22, 2021 • edited by alalek Loading

Pull Request Readiness Checklist

alalek left a comment

Choose a reason for hiding this comment

jumpy88 commented Feb 26, 2021

komakai commented Feb 27, 2021

jumpy88 commented Mar 5, 2021

jumpy88 commented Mar 25, 2021 • edited Loading

alalek Mar 25, 2021

Choose a reason for hiding this comment

komakai Mar 26, 2021

Choose a reason for hiding this comment

splintersilk Mar 26, 2021

Choose a reason for hiding this comment

Ay-De commented Feb 9, 2022

Ay-De commented Feb 11, 2022

komakai commented Feb 12, 2022

Ay-De commented Feb 12, 2022

komakai commented Feb 13, 2022

Ay-De commented Feb 13, 2022 • edited Loading

komakai commented Feb 14, 2022

Ay-De commented Feb 14, 2022

komakai commented Feb 15, 2022

Ay-De commented Feb 15, 2022 • edited Loading

komakai commented Feb 16, 2022

Ay-De commented Feb 20, 2022

asadasivam commented Mar 8, 2023

komakai commented Feb 22, 2021 •

edited by alalek

Loading

jumpy88 commented Mar 25, 2021 •

edited

Loading

Ay-De commented Feb 13, 2022 •

edited

Loading

Ay-De commented Feb 15, 2022 •

edited

Loading