🐛 Android frame processor not works

[luisfuertes]

What were you trying to do?

I have installed example project and my own project with a fame processor.

On example project frame processor dont have class "QRCodeFrameProcessorPluginPackage" (in the documentation it says that you have to add it), it only have a "ExampleFrameProcessorPlugin"

On MainApplication.java example project add CameraPackage on packages list

packages.add(new CameraPackage());

And in onCreate add the module:

FrameProcessorPlugin.register(new ExampleFrameProcessorPlugin());

In documentation it says that you have to add your custom frame processor package class (QRCodeFrameProcessorPluginPackage in documentation example) to packages list.

I can build correctly Example project and my own project (in my project I have created the frame processor as it says in the documentation with npx vision-camera-plugin-builder android script).

They both compile but neither of them seem to call the frame processor. At least it doesn't show any errors or the log that the example frame processor should show in javascript.

Any suggestion?

Reproduceable Code

Example project or new project using create frame processor script on android

What happened instead?

They both compile but neither of them seem to call the frame processor. At least it doesn't show any errors or the log that the example frame processor should show in javascript.

Relevant log output

Android logcat when i execute code (the app works and the camera is displayed, but the frame processor is not called)

07-08 18:08:54.664   768  3695 I mm-camera: <IFACE >< INFO> 2423: iface_axi_handle_sof_event: Event SOF session 4 VFE1, src 0 with frame_id 10859 ts: 2680067017886000
07-08 18:08:54.682   723  4312 E ResolverController: No valid NAT64 prefix (102, <unspecified>/0)
07-08 18:08:54.698   768  3695 I mm-camera: <IFACE >< INFO> 2423: iface_axi_handle_sof_event: Event SOF session 4 VFE1, src 0 with frame_id 10860 ts: 2680067051218000
07-08 18:08:54.732   768  3695 I mm-camera: <IFACE >< INFO> 2423: iface_axi_handle_sof_event: Event SOF session 4 VFE1, src 0 with frame_id 10861 ts: 2680067084543000
07-08 18:08:54.733   768  3656 I mm-camera: <ISP   >< INFO> 245: ihist_stats46_stats_config_validate: warning: Invalid IHIST ROI from 3A 0 0 0 0
07-08 18:08:54.767   768  3695 I mm-camera: <IFACE >< INFO> 2423: iface_axi_handle_sof_event: Event SOF session 4 VFE1, src 0 with frame_id 10862 ts: 2680067117863000
07-08 18:08:54.802   768  3695 I mm-camera: <IFACE >< INFO> 2423: iface_axi_handle_sof_event: Event SOF session 4 VFE1, src 0 with frame_id 10863 ts: 2680067151186000

Device

Xiaomi Mi A2 Android 10

VisionCamera Version

^2.13.5

Additional information

• I am using Expo
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[luisfuertes]

My camera component

const ComponentView: FC<Props> = () => {
  const cameraRef = useRef<Camera>(null);
  const [isCameraInitialized, setIsCameraInitialized] = useState(false);

  // check if camera page is active
  const isFocussed = useIsFocused();
  const isForeground = useIsForeground();
  const isActive = isFocussed && isForeground;

  // check devices
  const [cameraPosition, setCameraPosition] = useState<'front' | 'back'>('back');
  const devices = useCameraDevices();
  const device = devices[cameraPosition];

  // camera calbacks
  const onError = useCallback((error: CameraRuntimeError) => {
    console.error('onError: ', error);
  }, []);

  const onInitialized = useCallback(() => {
    console.log('Camera initialized!');
    setIsCameraInitialized(true);
  }, []);

  const frameProcessor = useFrameProcessor(frame => {
    'worklet';
    const values = examplePlugin(frame);
    console.log(`Return Values: ${JSON.stringify(values)}`);
  }, []);

  const onFrameProcessorSuggestionAvailable = useCallback((suggestion: FrameProcessorPerformanceSuggestion) => {
    console.log(`Suggestion available! ${suggestion.type}: Can do ${suggestion.suggestedFrameProcessorFps} FPS`);
  }, []);

  // TODO: ADD LOADING VIEW
  if (device == null) return;

  return (
    <SafeAreaView style={styles.container}>
      <Camera
        style={styles.camera}
        device={device}
        isActive={isActive}
        onInitialized={onInitialized}
        onError={onError}
        enableZoomGesture={false}
        frameProcessor={device.supportsParallelVideoProcessing ? frameProcessor : undefined}
        onFrameProcessorPerformanceSuggestionAvailable={onFrameProcessorSuggestionAvailable}
        frameProcessorFps={1}
        photo={false}
        audio={false}
        video
      />
    </SafeAreaView>
  );
};

export default ComponentView;

My frame processor on javascript

import type { Frame } from 'react-native-vision-camera';

declare let _WORKLET: true | undefined;

export function examplePlugin(frame: Frame): string[] {
  'worklet';
  if (!_WORKLET) throw new Error('examplePlugin must be called from a frame processor!');

  // @ts-expect-error because this function is dynamically injected by VisionCamera
  return __examplePlugin(frame, 'hello!', 'parameter2', true, 42, { test: 0, second: 'test' }, ['another test', 5]);
}

And my custom frame processor FrameProcessorPlugin.java

package com.example.exampleframeprocessor;

import android.util.Log;
import androidx.camera.core.ImageProxy;
import com.facebook.react.bridge.WritableNativeArray;
import com.facebook.react.bridge.WritableNativeMap;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;
import org.jetbrains.annotations.NotNull;


public class ExampleFrameProcessorPlugin extends FrameProcessorPlugin {
  @Override
  public Object callback(@NotNull ImageProxy image, @NotNull Object[] params) {
    Log.v("ExampleFrameProcessor", image.getWidth() + " x " + image.getHeight() + " Image with format #" + image.getFormat() + ". Logging " + params.length + " parameters:");

    for (Object param : params) {
      Log.v("ExampleFrameProcessor", "  -> " + (param == null ? "(null)" : param.toString() + " (" + param.getClass().getName() + ")"));
    }

    WritableNativeMap map = new WritableNativeMap();
    map.putString("example_str", "Test");
    map.putBoolean("example_bool", true);
    map.putDouble("example_double", 5.3);

    WritableNativeArray array = new WritableNativeArray();
    array.pushString("Hello!");
    array.pushBoolean(true);
    array.pushDouble(17.38);

    map.putArray("example_array", array);
    return map;
  }

  public ExampleFrameProcessorPlugin() {
    super("examplePlugin");
  }
}

And my ExampleFrameProcessorPluginPackage

package com.example.exampleframeprocessor;

import androidx.annotation.NonNull;

import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;

import java.util.Collections;
import java.util.List;

public class ExampleFrameProcessorPluginPackage implements ReactPackage {
  @NonNull
  @Override
  public List<NativeModule> createNativeModules(@NonNull ReactApplicationContext reactContext) {
    FrameProcessorPlugin.register(new ExampleFrameProcessorPlugin
            ());
    return Collections.emptyList();
  }

  @NonNull
  @Override
  public List<ViewManager> createViewManagers(@NonNull ReactApplicationContext reactContext) {
    return Collections.emptyList();
  }
}

I can see log Camera initialized! but dont show any error log or frame processor log on javascript

PD: Example project and my own project works fine on iOS

[mrousavy]

Hey! I've rewritten the entire Android codebase of VisionCamera from CameraX to Camera2 in the efforts of ✨ VisionCamera V3.

I just now completed the Camera2 rewrite and I believe the core structure is running, but there might be some edge cases to iron out. Can you try and test the PR #1674 for me to see if you can still reproduce this issue here?

Here's an instruction on how you can test that: #1674 (comment)

If the issue cannot be reproduced with that version/PR anymore, then hoorayy, I fixed it! 🎉
Otherwise please let me know and I'll keep this issue open to keep track of it.

Thank you!