Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: libcamera support #519

Open
rothn opened this issue Sep 12, 2022 · 85 comments
Open

Feature Request: libcamera support #519

rothn opened this issue Sep 12, 2022 · 85 comments

Comments

@rothn
Copy link

rothn commented Sep 12, 2022

Is your feature request related to a problem? Please describe.
Waydroid currently supports mainline Linux cameras by passing low-level V4L2 camera devices through to Android, which sets them up as extremely limited external webcams. This is not ideal, e.g. for Android apps that use the front-facing camera to capture QR codes.

Describe the solution you'd like

  • Include libcamera as an optional dependency
  • Install the compiled libcamera Android HAL in the container
  • Set "ro.hardware.camera=<libcamera-so-path>" instead of "ro.hardware.camera=v4l2" on supported hardware

Describe alternatives you've considered

  • One-off camera implementations per platform: Infeasible due to required effort, but support would be betteer
  • Vanilla v4l2-- Generally unusable because lacks color calibration support, autofocus, etc. found in complex cameras (e.g., PinePhone Pro, Librem 5)

Additional context

@rothn
Copy link
Author

rothn commented Sep 13, 2022

@aleasto @erfanoabdi I'm probably going to try to knock these out over the weekend. Does the scope I suggested under "Describe the solution you'd like" sound right to you? Anything I'm missing that I may want to consider?

@aleasto
Copy link
Member

aleasto commented Sep 14, 2022

libcamera already implements the android HAL interface?

@rothn
Copy link
Author

rothn commented Sep 14, 2022

Yes tree link, although it's based on camera3.h, which is only supported on Android < 8 according to the docs. So part of the scope might also have to include porting libcamera's HAL linkage from standard C++ to AIDL with XML metadata. Not a weekend project anymore.

@aleasto
Copy link
Member

aleasto commented Sep 14, 2022

No it's ok. The default hidl implementations already wrap around the legacy hal format. Everybody uses that.

@aleasto
Copy link
Member

aleasto commented Sep 14, 2022

However it doesn't look like libcamera builds inline with the aosp build system

@rothn
Copy link
Author

rothn commented Sep 14, 2022

No, it doesn't. Think that will be an issue? I was thinking I'd build a .so file, find the right place to drop it (e.g. /vendor), and hopefully it would be picked up correctly.

@aleasto
Copy link
Member

aleasto commented Sep 14, 2022

The right place would be /vendor/lib[64]/hw/camera.libcamera.so and ro.hardware.camera=libcamera

There's a way to include a prebuilt blob in aosp but it's less than ideal. We don't do this for any other native component.

You also need to figure out how to build against the android NDK rather than your host's glibc & co. Since the android part seems to be supported upstream, they might already have tools for this.

@aleasto
Copy link
Member

aleasto commented Sep 14, 2022

kbingham/libcamera#26
https://github.com/GloDroid/glodroid_forks/commits/libcamera-next

Here's an effort to build inline with aosp following the same approach as freedesktop/mesa

@rothn
Copy link
Author

rothn commented Oct 20, 2022

@aleasto I'm stumped... I've got libcamera building inline now and updated Waydroid to use libcamera (see WIP section of this guide.

The files are there:

  • /vendor/lib64/hw/camera.libcamera.so
  • /vendor/lib64/libcamera.so
  • /vendor/lib64/libcamera-base.so

And yet, according to Android, based on lshal and dumpsys media.camera, this HAL does not exist!

EDIT: I was able to get a little farther, to where the .so seems to load but still no cameras:

10-20 00:44:41.142    71    71 I CamPrvdr@2.4-legacy: Loaded "libcamera camera HALv3 module" camera module
10-20 00:44:41.142    71    71 I CamPrvdr@2.4-legacy: setUpVendorTags: No vendor tags defined for this device.

I'm working with the libcamera team to support a full inline build, but I need to be able to validate that this works first. Any ideas or guesses as to what I might be missing? I'm really stumped and would appreciate your help.

libcamera_debug.log
logcat.log
logcat.log

Thanks in advance!

@aleasto
Copy link
Member

aleasto commented Oct 20, 2022

It looks correctly hooked up.

BTW I'm not sure anybody has verified that the camera stack still works in lineage-18.1 builds. Have you verified on a compatible camera that the current V4L2 hal works prior to this work?

@kbingham
Copy link

Can you try to set LIBCAMERA_LOG_LEVELS=:0 as an environment variable for whatever process space is loading the libcamera HAL module please? (and share the logs)

@rothn
Copy link
Author

rothn commented Oct 20, 2022

@aleasto The V4L2 HAL doesn't work, which isn't surprising. The RK3399 SoC my phone is based on is kind of weird in that it needs a userland control loop to really work. libcamera has a special pipeline (rkisp2 IIRC) that implements this.

Though the camera technically implements V4L2, a quick read-through of Google's V4L2 implementation and TODO(b/*) comments makes it pretty clear that Google's V4L2 HAL will reject cameras from that SoC.

@aleasto
Copy link
Member

aleasto commented Oct 20, 2022

on a compatible camera was the key there

@rothn
Copy link
Author

rothn commented Oct 20, 2022

@kbingham I believe that the libcamera logger is broken, and have filed a ticket with more details: https://bugs.libcamera.org/show_bug.cgi?id=161
Working around that, I get a much more informative logcat: logcat.log

It looks like this expects (but does not find) /dev/media[0-3], which looks like a pretty good clue. Investigating...

Key output:

10-20 14:57:11.467    76    76 W android.hardware.camera.provider@2.4-service: LIBCAMERA_LOG_LEVELS is 0
10-20 14:57:11.467    76    76 W android.hardware.camera.provider@2.4-service: Initialising Android camera HAL
10-20 14:57:11.467    76    76 W android.hardware.camera.provider@2.4-service: Failed to open configuration file /vendor/etc/libcamera/camera_hal.yaml: No such file or directory
10-20 14:57:11.468    76    76 W android.hardware.camera.provider@2.4-service: Public key not valid
10-20 14:57:11.468    76    76 W android.hardware.camera.provider@2.4-service: No IPA found in '/usr/local/lib/libcamera'
10-20 14:57:11.468    76    76 W android.hardware.camera.provider@2.4-service: libcamera v0.0.1
10-20 14:57:11.469    76   102 W android.hardware.camera.provider@2.4-service: Starting camera manager
10-20 14:57:11.470    76   102 W android.hardware.camera.provider@2.4-service: Device node /dev/media2 should exist but doesn't
10-20 14:57:11.470    76   102 W android.hardware.camera.provider@2.4-service: Device node /dev/media0 should exist but doesn't
10-20 14:57:11.470    76   102 W android.hardware.camera.provider@2.4-service: Device node /dev/media3 should exist but doesn't
10-20 14:57:11.471    76   102 W android.hardware.camera.provider@2.4-service: Device node /dev/media1 should exist but doesn't
10-20 14:57:11.471    76   102 W android.hardware.camera.provider@2.4-service: Found registered pipeline handler 'SimplePipelineHandler'
10-20 14:57:11.471    76   102 W android.hardware.camera.provider@2.4-service: Found registered pipeline handler 'PipelineHandlerRkISP1'
10-20 14:57:11.472    76    76 I CamPrvdr@2.4-legacy: Loaded "libcamera camera HALv3 module" camera module
10-20 14:57:11.472    76    76 I CamPrvdr@2.4-legacy: setUpVendorTags: No vendor tags defined for this device.
10-20 14:57:11.472   101   101 I HidlServiceManagement: Registered android.hardware.gatekeeper@1.0::IGatekeeper/default
10-20 14:57:11.474   101   101 I HidlServiceManagement: Removing namespace from process name android.hardware.gatekeeper@1.0-service to gatekeeper@1.0-service.
10-20 14:57:11.474   101   101 I LegacySupport: Registration complete for android.hardware.gatekeeper@1.0::IGatekeeper/default.
10-20 14:57:11.475    76    76 I HidlServiceManagement: Registered android.hardware.camera.provider@2.4::ICameraProvider/legacy/0
10-20 14:57:11.475    76    76 I HidlServiceManagement: Removing namespace from process name android.hardware.camera.provider@2.4-service to provider@2.4-service.
10-20 14:57:11.476    76    76 I LegacySupport: Registration complete for android.hardware.camera.provider@2.4::ICameraProvider/legacy/0.
...
10-20 14:57:13.499   144   144 I CameraProviderManager: Connecting to new camera provider: legacy/0, isRemote? 1
10-20 14:57:13.500   144   144 I CameraProviderManager: Camera provider legacy/0 ready with 0 camera devices
10-20 14:57:13.500   144   182 W CameraProviderManager: addProviderLocked: Camera provider HAL with name 'legacy/0' already registered
10-20 14:57:13.503   144   144 I cameraserver: Waiting for activity service

@aleasto
Copy link
Member

aleasto commented Oct 20, 2022

Add the devices to /var/lib/waydroid/lxc/waydroid/config_nodes
We'll have to hook that in.

@kbingham
Copy link

Is waydroid containerised? Does it need to pass through access to the video and media nodes? Or all of /dev?

@aleasto
Copy link
Member

aleasto commented Oct 20, 2022

Is waydroid containerised? Does it need to pass through access to the video and media nodes? Or all of /dev?

It is! We bind mount only the useful dev nodes

@rothn
Copy link
Author

rothn commented Oct 20, 2022

@aleasto New errors, any ideas what I need to set to fix the permissions on /dev/media[0-3]?

10-20 15:27:46.158    69   150 W android.hardware.camera.provider@2.4-service: Failed to open media device at /dev/media2: Permission denied
10-20 15:27:46.158    69   150 W android.hardware.camera.provider@2.4-service: Unable to populate media device /dev/media2 (Permission denied), skipping
10-20 15:27:46.158    69   150 W android.hardware.camera.provider@2.4-service: Failed to open media device at /dev/media0: Permission denied
10-20 15:27:46.158    69   150 W android.hardware.camera.provider@2.4-service: Unable to populate media device /dev/media0 (Permission denied), skipping
10-20 15:27:46.158    69   150 W android.hardware.camera.provider@2.4-service: Failed to open media device at /dev/media3: Permission denied
10-20 15:27:46.158    69   150 W android.hardware.camera.provider@2.4-service: Unable to populate media device /dev/media3 (Permission denied), skipping
10-20 15:27:46.159    69   150 W android.hardware.camera.provider@2.4-service: Failed to open media device at /dev/media1: Permission denied
10-20 15:27:46.159    69   150 W android.hardware.camera.provider@2.4-service: Unable to populate media device /dev/media1 (Permission denied), skipping

@aleasto
Copy link
Member

aleasto commented Oct 20, 2022

Looks like for other nodes we chmod 777.
You might also be constrained by selinux if your host system uses that.

@rothn
Copy link
Author

rothn commented Oct 20, 2022

Sounds good. I've got an active waydroid fork too now :).

Looks like I'll need to do just a bit more work to test this, likely creating some sort of config file for libcamera

EDIT: This seems like a subset of the errors from kbingham/libcamera#28 and matches
rafael2k@'s observed errors ("Unable to get rectangle [0-2] on pad 0"). Diving in...

EDIT: This was a red herring.

My kernel: 5.19.8-1-MANJARO-ARM

10-20 16:46:08.398    79    79 W android.hardware.camera.provider@2.4-service: libcamera v0.0.1
10-20 16:46:08.399    79   113 W android.hardware.camera.provider@2.4-service: Starting camera manager
10-20 16:46:08.400    79   113 W android.hardware.camera.provider@2.4-service: New media device "rkisp1" created from /dev/media2
10-20 16:46:08.406    79   113 W android.hardware.camera.provider@2.4-service: Added device /dev/media2: rkisp1
10-20 16:46:08.407    79   113 W android.hardware.camera.provider@2.4-service: New media device "hantro-vpu" created from /dev/media0
10-20 16:46:08.408    79   113 W android.hardware.camera.provider@2.4-service: Added device /dev/media0: hantro-vpu
10-20 16:46:08.409    79   113 W android.hardware.camera.provider@2.4-service: New media device "rkisp1" created from /dev/media3
10-20 16:46:08.413    79   113 W android.hardware.camera.provider@2.4-service: Added device /dev/media3: rkisp1
10-20 16:46:08.413    79   113 W android.hardware.camera.provider@2.4-service: New media device "rkvdec" created from /dev/media1
10-20 16:46:08.414    79   113 W android.hardware.camera.provider@2.4-service: Added device /dev/media1: rkvdec
10-20 16:46:08.415    79   113 W android.hardware.camera.provider@2.4-service: Found registered pipeline handler 'SimplePipelineHandler'
10-20 16:46:08.415    79   113 W android.hardware.camera.provider@2.4-service: Found registered pipeline handler 'PipelineHandlerRkISP1'
10-20 16:46:08.415    79   113 W android.hardware.camera.provider@2.4-service: Successful match for media device "rkisp1"
10-20 16:46:08.416    79   113 W android.hardware.camera.provider@2.4-service: Opened device platform:rkisp1: rkisp1: rkisp1_stats
10-20 16:46:08.416    79   113 W android.hardware.camera.provider@2.4-service: Opened device platform:rkisp1: rkisp1: rkisp1_params
10-20 16:46:08.416    79   113 W android.hardware.camera.provider@2.4-service: Opened device platform:rkisp1: rkisp1: rkisp1
10-20 16:46:08.417    79   113 W android.hardware.camera.provider@2.4-service: Opened device platform:rkisp1: rkisp1: rkisp1
10-20 16:46:08.417    79   113 W android.hardware.camera.provider@2.4-service: Control: Exposure (0x00980911)
10-20 16:46:08.418    79   113 W android.hardware.camera.provider@2.4-service: Control: Vertical Blanking (0x009e0901)
10-20 16:46:08.418    79   113 W android.hardware.camera.provider@2.4-service: Control: Horizontal Blanking (0x009e0902)
10-20 16:46:08.418    79   113 W android.hardware.camera.provider@2.4-service: Control: Analogue Gain (0x009e0903)
10-20 16:46:08.418    79   113 W android.hardware.camera.provider@2.4-service: Control: Link Frequency (0x009f0901)
10-20 16:46:08.418    79   113 W android.hardware.camera.provider@2.4-service: Control: Pixel Rate (0x009f0902)
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: Control: Test Pattern (0x009f0903)
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: Control: Digital Gain (0x009f0905)
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: Optional V4L2 control 0x009a0923 not supported
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: Recommended V4L2 control 0x009a0922 not supported
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: Unable to get rectangle 2 on pad 0: Inappropriate ioctl for device
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: The PixelArraySize property has been defaulted to 3264x2448
10-20 16:46:08.419    79   113 W android.hardware.camera.provider@2.4-service: Unable to get rectangle 1 on pad 0: Inappropriate ioctl for device
10-20 16:46:08.420    79   113 W android.hardware.camera.provider@2.4-service: The PixelArrayActiveAreas property has been defaulted to (0, 0)/3264x2448
10-20 16:46:08.420    79   113 W android.hardware.camera.provider@2.4-service: Unable to get rectangle 0 on pad 0: Inappropriate ioctl for device
10-20 16:46:08.420    79   113 W android.hardware.camera.provider@2.4-service: Failed to retrieve the sensor crop rectangle
10-20 16:46:08.420    79   113 W android.hardware.camera.provider@2.4-service: The sensor kernel driver needs to be fixed
10-20 16:46:08.420    79   113 W android.hardware.camera.provider@2.4-service: See Documentation/sensor_driver_requirements.rst in the libcamera sources for more information
10-20 16:46:08.421    79   113 W android.hardware.camera.provider@2.4-service: No static properties available for 'm00_f_ov8858'
10-20 16:46:08.421    79   113 W android.hardware.camera.provider@2.4-service: Please consider updating the camera sensor properties database
10-20 16:46:08.421    79   113 W android.hardware.camera.provider@2.4-service: Failed to retrieve the camera location
10-20 16:46:08.422    79   113 W android.hardware.camera.provider@2.4-service: Set a delay of 2 and priority write flag 0 for Exposure
10-20 16:46:08.422    79   113 W android.hardware.camera.provider@2.4-service: Set a delay of 1 and priority write flag 0 for Analogue Gain
10-20 16:46:08.422    79   113 W android.hardware.camera.provider@2.4-service: Releasing buffers
10-20 16:46:08.422    79   113 W android.hardware.camera.provider@2.4-service: 0 buffers requested.
10-20 16:46:08.423    79   113 W android.hardware.camera.provider@2.4-service: Releasing buffers
10-20 16:46:08.423    79   113 W android.hardware.camera.provider@2.4-service: 0 buffers requested.
10-20 16:46:08.423    79   113 W android.hardware.camera.provider@2.4-service: Releasing buffers
10-20 16:46:08.423    79   113 W android.hardware.camera.provider@2.4-service: 0 buffers requested.
10-20 16:46:08.423    79   113 W android.hardware.camera.provider@2.4-service: Releasing buffers
10-20 16:46:08.424    79   113 W android.hardware.camera.provider@2.4-service: 0 buffers requested.

@rothn
Copy link
Author

rothn commented Oct 21, 2022

So, it turns out the IPAs rely on IPC, which in turn relies on building a functioning executable. That's turning out to be tricky in this situation.

@kbingham
TL;DR: Disabling IPA isolation on Android would drastically simplify the design while solving a problem. Would you be open to this?

Long version: Would you be open to disabling IPA isolation on Android if we can't get this to work? Another option might be to use their build system. My best guess is that Android has some kind of special sauce in their CRT kind of like Microsoft's mainCRTStartup() business on Windows. They go to an awful lot of trouble to explicitly specify their CRT over the LLVM default (Reference: Android code search). Even if I do find a hack, I'm not sure it will be worth the added maintenance burden over disabling IPA isolation on Android, which would also solve the problem of having to build executable in Android.mk too.

64-bit rkisp1_ipa_proxy_worker

[1:59:59.451846088] [3454] ERROR IPAProxyRkISP1Worker rkisp1_ipa_proxy_worker.cpp:471 argc: -536842124
[1:59:59.452267235] [3454] ERROR IPAProxyRkISP1Worker rkisp1_ipa_proxy_worker.cpp:473 argc: -536842124
[1:59:59.452492682] [3454] ERROR IPAProxyRkISP1Worker rkisp1_ipa_proxy_worker.cpp:475 argv[0]: �p���
[1:59:59.452672049] [3454] ERROR IPAProxyRkISP1Worker rkisp1_ipa_proxy_worker.cpp:479 Tried to start worker with no args: expected <path to IPA so> <fd to bind unix socket>
Bus error (core dumped)

32-bit rkisp1_ipa_proxy_worker

139|:/ # /data/local/tmp/rkisp1_ipa_proxy foo 35                                                                                                                                                                                                                                                     
[2:07:07.094747353] [3550] ERROR IPAProxyRkISP1Worker rkisp1_ipa_proxy_worker.cpp:471 argc: 9743981
[2:07:07.095134670] [3550] ERROR IPAProxyRkISP1Worker rkisp1_ipa_proxy_worker.cpp:473 argc: 9743981
Segmentation fault (core dumped)

@kbingham
Copy link

It's easy to disable IPA isolation in your build to get this working. I think rsglobal what had a patch doing that. If you can't find it let me know and I'll dig out the pointers.

Isolation can be disabled, but It's always up to the distribution to deal with if isolation is used or not, and how. Right now, you're the distribution, so you can just hack it out. The IPA still has to be loaded via dlopen though

@rothn
Copy link
Author

rothn commented Oct 23, 2022

@kbingham , I posted a patch file and logs to the development list. Looking forward to reading what folks have to say. Looks like the message is being held for moderation since the logs are long.

@aleasto and others, I need your help with validation (trying out the HAL)! I've documented my setup here, with libcamera-specific changes under the heading "[WIP] Camera Support for PinePhone Pro". I'll be transparent-- on the PinePhone Pro, this only somewhat works. I see the camera devices correctly in an Android diagnostic data app, but the Java camera2 implementation raises an exception when apps try to use it. I'm not sure whether this is a problem with the rkisp1 ISP (Rockchip, as in the PinePhone Pro's SOC) or with libcamera's Android HAL. It will help me debugging to learn if this works for other devices, if this is broken in the same way for all devices, or broken in different ways for different devices.

@darkdragon-001
Copy link
Contributor

@rothn I remember megi working on camera support for Pinephone Pro and I am not sure how much is missing on mainstream kernel / libcamera side. Here is his latest blog post regarding this topic: https://xnux.eu/log/#070
Did you try to contact him?

@kbingham
Copy link

Thanks ok I see it on the list. I'll reply later or tomorrow as I'm away from home currently. It looks like you've squashed everything into a single patch, which is against our code development practices, so there's a fair bit of work for cleanup. But let's focus on getting it working first.

@rothn
Copy link
Author

rothn commented Oct 23, 2022

@kbingham Fundamentally, it does seem to work -- based on the logs it just seems like camera2 doesn't like the provided video modes from the rkisp1 ISP + my image sensors + how the HAL provides video modes. I would not be surprised if Android camera2 works usefully on top of this for other devices with different ISPs and sensors. I'm going to continue trying to get my hardware configuration working, but I'd like to keep submitting small incremental improvements rather than have a gigantic review once the whole thing works just how I want it to, although of course that's ultimately up to you. Also, see email on your list for how I suggest splitting the patch and LMK if that looks good to you.

@darkdragon-001 This builds on top of megi@'s mainline kernel driver, so in a way he made this possible. I shot him an email asking for feedback.

@rothn
Copy link
Author

rothn commented Oct 23, 2022

@darkdragon-001
I reached out to megi and it seems like his work would complement this well, especially by adding calibration profiles for the PinePhone Pro's cameras, and perhaps letting people calibrate their own.

To paraphrase, he's aware of libcamera, though he notes that the rkisp1 backend could use auto white-balance, per-light source color profiles, and light source detection.

He's done some work on calibration tools (which I would welcome since my hardware defaults to "uncalibrated.yaml" in libcamera), and though that's on hold for now he expects to pick it back up early next year.

@aleasto
Copy link
Member

aleasto commented Oct 23, 2022

@kbingham Fundamentally, it does seem to work -- based on the logs it just seems like camera2 doesn't like the provided video modes from the rkisp1 ISP + my image sensors + how the HAL provides video modes. I would not be surprised if Android camera2 works usefully on top of this for other devices with different ISPs and sensors. I'm going to continue trying to get my hardware configuration working, but I'd like to keep submitting small incremental improvements rather than have a gigantic review once the whole thing works just how I want it to, although of course that's ultimately up to you.

You should also try ro.hardware.gralloc=minigbm_gbm_mesa

@kbingham
Copy link

Smaller commits posted incrementally as soon as they are ready is certainly more appreciated. Your latest series is working in that direction, so that's much more helpful thanks.

Regarding

"based on the logs it just seems like camera2 doesn't like the provided video modes from the rkisp1 ISP + my image sensors + how the HAL provides video modes. I would not be surprised if Android camera2 works usefully on top of this for other devices with different ISPs and sensors. "

The RKISP1 is used in Chrometabs, and is expected to function through the android HAL, though it's not been tested enough in libcamera. But you should certainly expect that it should be possible to operate successfully.

@rothn
Copy link
Author

rothn commented Dec 2, 2022

@rmader Writing an android camera HAL is a lot of work, and libcamera already has a presumably full-featured one that needs some minor fixes to build inline with AOSP for. I’ve looked into writing one, and it’s quite a lot of effort.

I’d be happy to provide help where I can if someone would be interested in doing this, but I don’t have the time unfortunately. Do you?

@rmader
Copy link

rmader commented Dec 3, 2022

I’d be happy to provide help where I can if someone would be interested in doing this, but I don’t have the time unfortunately. Do you?

Same here, unfortunately not, I just got side-tracked into camera stuff. But I'm currently trying to look into some gaps in Pipewire which will hopefully help in case you'll need to switch approaches at some point (most importantly the 90 deg. camera rotations in many phones). In any case, much appreciate your work, thanks a lot!

@kbingham
Copy link

kbingham commented Dec 3, 2022

Indeed, I don't quite see pipewire fitting here exactly yet. Android expects a specific HAL API implementing, with direct control over the camera, which pipewire itself is a long way from supporting.

I won't say never though

@jmondi
Copy link

jmondi commented Dec 3, 2022

I'm not up to speed with Android internals, but as far as I get it there's no role for anything like Pipewire in Android. The Android frameworks for audio/video (as well as all others I presume) are designed as vertical components that span from high level application-facing APIs to the "platform" adaption code, and their implementation is core Android stuff which seems rather hard to replace or plumb into. Unless there are already plans to move Android to use pipewire, which I'm not aware of at the moment

@Quackdoc
Copy link
Contributor

Quackdoc commented Dec 3, 2022

@jmondi the goal would be to interface pipewire with the camera HAL, which to be clear, I think is the right solution in the future.

as far as I am aware, libcamera works exclusively and doesn't offer multiplexing right? one gets access at a time? and the goal of pipewire here is to mediate control. I think this means that the host would not have access to the camera when waydroid is running. this poses a set of issues that gets solved via pipewire.

I would argue that pipewire is the "best solution" in this regard, however until pipewire offers all the direct controls that would need to be exposed, it's pretty much a non starter.

the only other solution that I can see as viable would be to somehow get v4l2loopback working, and use that as a bridge to feed libcamera to android, however we this would present a whole slew of drawbacks.

so while I think pipewire is the route forward, until it gets the functionality it needs libcamera passthrough seems like the best route.

@rmader
Copy link

rmader commented Dec 4, 2022

@rothn: regarding your kernel work on ov8858/imx258 for the Pinephone Pro, one open issue I see is that the 90 degree rotation is not yet detected/reported in CameraConfiguration::transform. Did you happen to look into this already / do you maybe plan to? (I'd be super happy if you did :P)

Context here is that I just managed to implement support for rotations in Pipewire but needed to override the value in libcamera.

Edit: I assume this is something needed for Waydroid as well.

@rothn
Copy link
Author

rothn commented Dec 4, 2022 via email

@jmondi

This comment was marked as off-topic.

@jmondi
Copy link

jmondi commented Dec 5, 2022

I probably just realized your concern is arbitration between the container and the host, not within the container itself ?

Then I have to ask: how does it work for other subsystems, in example audio ? If an application running in the container plays music, do you expect a native application running on the Linux host to be able to access the same audio device at the same time ? Why are cameras any different ?

If an application running in the container accesses the camera, how could another application, running in a different context space access the same hw device at the same time ? What if they want to stream different formats/resolutions concurrently ?

Pipewire handles multiplexing at an higher level, by sharing references to the video buffers produced by the camera between different applications, and it possibly applies transformation by piping the buffers to other components (encoders, display, scalers etc) but it doesn't multiplex access to the camera HW configuration which is unique and constant per each streaming session. The "media session daemon" role in Android is played by the Camera Service, probably not in the same way as Pipewire does, as Android defines APIs for application to handle post-processing/endcoding/displaying by themselves, as applications, by definition, realize a specific use case by bundling together a set of system resources they then manage and combine together.

As I understand it, if you expose you're media devices to the container and the container uses them, the host won't be able to access the same devices concurrently, but this doesn't seem different than any other subsystem like audio (graphics I assume it's a bit more complicated as it certainly has to be shared between the container and the host, but I presume this is handled by lxc maybe ?)

@jmondi
Copy link

jmondi commented Dec 5, 2022

@rothn @rmader If Robert is asking "does your ov8858 driver register the V4L2_CID_ROTATION and V4L2_CID_ORIENTATION" controls (from which libcamera parses properties::Rotation and properties::Location) I guess the answer is that the first submission by Nicholas doesn't but all it takes for the controls to be parsed and registered by a sensor driver is
https://lore.kernel.org/all/20200509090456.3496481-13-jacopo@jmondi.org/

Also if there are any comments on the driver send them to the linux-media mailing list pretty please, not to leave them behind in the github walled garden

@rmader
Copy link

rmader commented Dec 5, 2022

If Robert is asking "does your ov8858 driver register the V4L2_CID_ROTATION and V4L2_CID_ORIENTATION" controls ...

That's exactly what I was trying to say and what I'd love to see added :)

Edit: from the libcamera side https://patchwork.libcamera.org/patch/17894/ is also needed IIUC.

@Quackdoc
Copy link
Contributor

Quackdoc commented Dec 5, 2022

Then I have to ask: how does it work for other subsystems, in example audio ? If an application running in the container plays music, do you expect a native application running on the Linux host to be able to access the same audio device at the same time ? Why are cameras any different ?

yes? If the host couldn't play audio at the same time, that would be a major flaw. I currently use waydroid to play some android games while watching video on the host, audio works there too. I know plenty of people that would like to use the camera's simultaneously, in fact, this is already possible on linux side, I personally do this with pw-v4l2 as it lets me record and stream on discord (through firefox) at the same time.

graphics, audio, input etc can all be used by the guest and the host at the same time (Input SHOULDN'T be, but that's a known issue).

the reason why lxc is used is explicitly because it gives that level of cooperation between the the host and waydroid. there are plenty of people who use waydroid for the "full android environment" and people who only use one or two apps between waydroid, so if at all possible, if the waydroid and the host could use the camera at the same time, that would be greatly preferred.

I can't say whose usecase it would be to utilize android on the host and guest at the same time, I can think of a couple theoretical ones. but in the end, I think ultimately more flexibility is best

@jmondi
Copy link

jmondi commented Dec 6, 2022

yes? If the host couldn't play audio at the same time, that would be a major flaw. I currently use waydroid to play some android games while watching video on the host, audio works there too. I know plenty of people that would like to use the camera's simultaneously, in fact, this is already possible on linux side, I personally do this with pw-v4l2 as it lets me record and stream on discord (through firefox) at the same time.

I see, I was not expecting this, but I certainly can't disagree with "the more flexibility, the better".

On your linux side it's fine that you can multiplex access to your camera, as pipewire does the muxing for you, but in the end it's pipewire itself the sole user of the camera hw. Arbitrating that between the host and the container is something I can't currently picture in my mind but I understand would be desirable...

@jmondi
Copy link

jmondi commented Dec 6, 2022

Ok, I got a chat with other libcamera devs (@kbingham) and that's my current understanding

Basically, right now we could "easily" get here:

waydroid_1

While you would like this

waydroid_2

In this last case it means you have to write a camera HAL on top of pipewire, and even if that happens the camera configuration between the host and the container should be the same.

Hope this clarifies my understanding and sorry for having got you wrong in my first reply

@kbingham
Copy link

kbingham commented Dec 6, 2022

Thanks for the diagrams @jmondi !

Yes, I agree. Pipewire could certainly implement an Android HAL which would let the container talk to a pipewire service on the host, and would then talk to libcamera.

This would be a fair amount of work I expect, and my worry would be - what use case does it solve? Any camera application within the Waydroid would probably expect direct control of the camera, which means the stream can't be shared...

It would not be 'reasonable' for instance to have the host in a video call, and then for waydroid to want to take a still image capture with the same camera. That would require reconfiguration of the camera, to support the still image, and would be prohibited if the pipewire daemon was actively streaming to a video call.

the only use case I could imagine currently being feasible for multiple camera access is if there were a simultaneous video call on the host and within a camera app in the waydroid environment. That would only then support a single video stream, and not be possible to reconfigure.

So - to support that, you'll need a Pipewire camera HAL as above. .. but that's a lot of work for ... one use case, that I don't currently see being widely used.

Now - me not thinking up other use cases isn't a reason not to implement an Android HAL on top of pipewire - but ... it will require some effort to do so. So I understand the 'flexibility' is best, but - Someone will have to put in the time and effort to do the work here if that's what you really want. Otherwise, using the libcamera Android HAL should already work.

@Quackdoc
Copy link
Contributor

Quackdoc commented Dec 6, 2022

@jmondi thats correct, that would be the "Ideal" situation, @kbingham there are a few other potential use cases. possibly being streaming + augmented reality stuff etc. but even with them, I do agree that pipewire implementation is more work then what it is currently worth.

however I do think that this could benefit users other then waydroid down the line, (I myself was looking into, briefly AGL+Linux which could potentially reap benefits) it's out of scope for waydroid to worry about now anyways.

@kklem0
Copy link

kklem0 commented Dec 9, 2022

Ok, I got a chat with other libcamera devs (@kbingham) and that's my current understanding

Basically, right now we could "easily" get here:

waydroid_1

While you would like this

waydroid_2

In this last case it means you have to write a camera HAL on top of pipewire, and even if that happens the camera configuration between the host and the container should be the same.

Hope this clarifies my understanding and sorry for having got you wrong in my first reply

@wtay
The way I see it, there're 2 more options,

  1. deeper PipeWire libcamera integration so they go into the same box.
  2. route all libcamera api through PipeWire and back to the real libcamera, very much like pipewire-alsa which act as a frontend for the applications that try to access alsa directly.

@rmader
Copy link

rmader commented Dec 20, 2022

FYI: I opened https://github.com/megous/linux/pull/17 and https://github.com/megous/linux/pull/18 with kernel driver and dts changes for rotation discovery (also needs https://patchwork.libcamera.org/patch/17891/). I'm not sure if that's needed for the Waydroid work, but I guess in the long run it would be nice if rotation values are picked up from the kernel device tree instead of profile files.

@baptx
Copy link

baptx commented Aug 25, 2023

Original issue: #170
It would be nice to write a message there if you get a (virtual) camera working.

@rothn
Copy link
Author

rothn commented Aug 25, 2023

I was able to get a black-and-white image to show up by hacking the API but a lot of things (e.g. taking a picture in Snapchat) would just crash the whole camera stack. This is because instead of properly converting from the NV12 stream libcamera produces in its HAL, I was directly copying that into Waydroid's buffer. Waydroid's Android image uses Mesa for rendering, which doesn't support YUV or NV12. So the result was undefined behavior, and it kind of (barely) worked.

The code for all of this is still on my Github and I'd happily walk anyone interested through it.

Next steps would be to figure out how to tell the libcamera Android HAL to produce an RGB stream. Probably by adding a flag into libcamera that defines what "IMPLEMENTATION_DEFINED" means, since that's the format libcamera generates and just assumes is NV12. For Mesa, this should be an RGB format, and while building this inline in AOSP we should then pass the specific RGB format we intend to use to the HAL's BUILD file (Bazel) or Makefile (legacy make).

Additional Format Considerations
Some ISPs (camera image processing hardware) are not able to produce RGB output in all configurations. In these cases, we'll ideally want to only use software conversion as a last resort fallback and stick to fully accelerated pipelines where possible. Linux phones are an emerging use-case for Waydroid, and tend to be sensitive to resource-intensive operations like software conversion of video streams.

Specific to Pinephone/RKISP1
In order to implement an RGB output at mainpath resolutions, we would need to do software conversion from NV12 to RGB. Or, alternatively, we would have to find some way to set up either mainpath or selfpath depending on which resolution is requested and only do this conversion if it exceeds what selfpath can do. I'd rather use mainpath for capturing stills since subjectively the processing quality looks better, but that could come later. Initially, I would recommend just figuring out how to configure the HAL to always use selfpath and limit it to that.

Let me know if you'd like to discuss more. I pushed this forward for awhile but got burnt out on this, perhaps bit off more than I could chew so to speak. I'd love to collaborate on this with someone who has the time this work deserves.

@baptx
Copy link

baptx commented Aug 25, 2023

@rothn Thanks for the update. I guess your fork https://github.com/rothn/waydroid also fails when trying to scan a QR code with WhatsApp to access the web version? Fortunately WhatsApp now added a way to access the web version without scanning a QR code.
Here is a reference of how it was done on Anbox with a fork by @alecbarber (maybe Anbox contributors will be interested to help with Waydroid now that Anbox is deprecated):
anbox/anbox#727 (comment)

@rothn
Copy link
Author

rothn commented Aug 25, 2023

Yes, it gets close though. Very close. The Uber Scooter app shows the preview window for capturing a barcode, but the camera stack crashes before recognizing the barcode. I suspect Uber and Snapchat both ask for a higher-quality stream for capture, and switching streams is just too many hacks on top of hacks for this stack.

@baptx
Copy link

baptx commented Aug 26, 2023

@rothn Do you think it would be possible to have camera support using the code from QEMU like done on the fork of Anbox or it would be more difficult than libcamera?

See this comment (anbox/anbox#727 (comment)):

I've implemented webcam support here: https://github.com/alecbarber/anbox/ for those who are happy to run Anbox manually (i.e. not as a snap). Using v4l2loopback it should be theoretically possible to pass a Whatsapp QR code into Anbox. I'm currently working on making the camera work when Anbox is packaged as a snap.

The basic idea is that Android/QEMU already comes with all the machinery required to pass the camera through (as alluded to by @NinoSkopac ), so the only thing that was needed was to plug the QEMU code into the existing Anbox session manager.

@rothn
Copy link
Author

rothn commented Aug 26, 2023

With a fake camera? Probably much easier than writing the Android HAL that libcamera wrote, but you'd have to write a HAL, which is probably more challenging than fixing configuration issues as making libcamera work would require.

@baptx
Copy link

baptx commented Aug 26, 2023

A virtual camera using v4l2loopback is better than nothing but someone said the Anbox fork even worked with a real webcam:
anbox/anbox#727 (comment)

@rothn
Copy link
Author

rothn commented Aug 26, 2023

I think what I'm trying to say is that it might be easier to reuse the existing real HAL than create a virtual camera HAL from scratch. But it looks like I might also be missing something important.

@DrYak
Copy link

DrYak commented Oct 9, 2023

Dear @rothn, had you had time to work further on supporting libcamera on Waydroid? I switched to a PinePhone Pro recently and am looking for a way to handle the applications which require QR-code for login.

rothn added a commit to rothn/libcamera that referenced this issue Oct 14, 2023
Currently, libcamera isolates any IPAs whose signatures cannot be
verified. Shared objects are created at build-time, and then signed. The
public signing key is embedded in a .cpp file, and libcamera verifies
IPA signatures at runtime. When libcamera cannot authenticate an IPA, it
runs it out-of-process.

This is problematic on three levels:
* IPA signing fundamentally does not work on Android for vendor modules
  like HALs (discussed below)
* Executables built to run out-of-process are not ABI-compatible with
  Android, making isolation infeasible [1]
* Linux phone hardware tends to be low-end because of the FOSS
  requirement, so the performance hit from out-of-process IPA isolation
  is significant

IPA signing fundamentally does not work for Android vendor modules:
After we "meson install" built .so files to a known location, Android
explicitly access them in PREBUILT_SHARED_LIBRARY or BUILD_PREBUILIT
to transform the .so files by stripping symbols among other things [2].
By modifying prebuilt libraries after we have already signed them, the
build system renders our signatures useless on Android.

Android distribution maintainers can use this flag to disable signature
verification, which will allow them to use libcamera.

[1] waydroid/waydroid#519
[2] https://cs.android.com/android/platform/superproject/+/master:build/make/core/cc_prebuilt_internal.mk?q=cc_prebuilt_internal

Signed-off-by: Nicholas Roth <nicholas@rothemail.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants