🐛[4.0.0-beta.15] AHardwareBuffer has unknown format (35) - cannot convert to SkImage!

[pweglik]

What's happening?

I get AHardwareBuffer has unknown format (35) - cannot convert to SkImage! error when trying to run basic example with skia frame processor. I found out it's related to pixel format:
https://developer.android.com/ndk/reference/group/a-hardware-buffer
RN Skia library expects type 1, and vision camera provides type 35.
The problem disappears when changeing pixelFormat to rgb in Camera props. Previously rgb was not working but maybe that changed in this version.

Android 12
RN Vision Camera 4.0.0-beta.15
"@shopify/react-native-skia": "^1.2.1"

Reproduceable Code

const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet';
    const paint = Skia.Paint();

    const blurFilter = Skia.ImageFilter.MakeBlur(10, 10, TileMode.Repeat, null);
    paint.setImageFilter(blurFilter);
    frame.render(paint);
  }, []);

<Camera
      style={StyleSheet.absoluteFill}
      device={device}
      isActive
      frameProcessor={frameProcessor}
      pixelFormat="yuv"
/>

Relevant log output

ERROR  Frame Processor Error: Exception in HostFunction: AHardwareBuffer has unknown format (35) - cannot convert to SkImage!, js engine: VisionCamera

Camera Device

{
  "formats": [],
  "sensorOrientation": "landscape-left",
  "hardwareLevel": "full",
  "maxZoom": 10,
  "minZoom": 1,
  "maxExposure": 24,
  "supportsLowLightBoost": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "isMultiCam": false,
  "minFocusDistance": 10,
  "minExposure": -24,
  "name": "0 (BACK) androidx.camera.camera2",
  "hasFlash": true,
  "hasTorch": true,
  "position": "back",
  "id": "0"
}

Device

Redmi Note Pro 9

VisionCamera Version

4.0.0-beta.15

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[pweglik]

Unfortunately, then resize plugin (https://github.com/mrousavy/vision-camera-resize-plugin) doesn't work: Frame Processor Error: Exception in HostFunction: java.lang.Error: Frame has invalid PixelFormat! Only YUV_420_888 is supported. Did you set pixelFormat="yuv"?, js engine: VisionCamera

[mrousavy]

Yea you need "rgb" for Skia on Android. On iOS, you can also use "yuv" (which is more efficient).

As for the resize-plugin; hm, this situation sucks a bit. I guess I can add rgb support to the resize plugin

[pweglik]

Hi, I'm not sure, why this was closed as completed as the bug is still present. I've just created PR in resize plugin repo solving it: mrousavy/vision-camera-resize-plugin#64

[mrousavy]

Hi, I'm not sure, why this was closed as completed as the bug is still present

That's a different bug - the issue is in Skia because Skia cannot work with YUV on Android yet. It needs to be RGB. On iOS, it can be both.

[pweglik]

Btw. I think the bug is fixed now. I checked code fro mthe description one more time and it works on Android now 😄

[tomerh2001]

Hi, I'm not sure, why this was closed as completed as the bug is still present

That's a different bug - the issue is in Skia because Skia cannot work with YUV on Android yet. It needs to be RGB. On iOS, it can be both.

Has this been fixed? Can we use YUV?