🐛 V3 runAsync doesn't work #1791

manolo-battista · 2023-09-11T15:39:37Z

What's happening?

On docs is described:

But I think it needs update, cause request frame as first parameter and func (callback) as second parameter.

The callback function is never called. I saw that code reach the func on runOnAsyncContext function, but I think enter always on finally cause I can't see my log ok 2.

Reproduceable Code

No response

Relevant log output

No response

Camera Device

No response

Device

iPhone 14 Pro

VisionCamera Version

3.0.0

Can you reproduce this issue in the VisionCamera Example app?

I can reproduce the issue in the VisionCamera Example app.

Additional information

I am using Expo
I have enabled Frame Processors (react-native-worklets-core)
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

The text was updated successfully, but these errors were encountered:

abdelrahman-muntaser · 2023-09-21T05:54:46Z

@manolo-battista did you find any solution for this ?

manolo-battista · 2023-09-21T07:59:49Z

@manolo-battista did you find any solution for this ?

no, unfortunately I still didn't find any solution. For my projects at the moment I'm using runAtTargetFps to run lower FPS rate.

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running synchronously at 60 FPS!")
  runAtTargetFps(2, () => {
    'worklet'
    console.log("I'm running synchronously at 2 FPS!")
  })
}, [])

mrousavy · 2023-09-30T10:14:02Z

Hi. Would've been great if you could've sent a PR for that to fix it in docs.

But in my tests, runAsync works - can you put a console.log in there to double check that it does not get called?

Can you reproduce it in the example app?

rodgomesc · 2023-10-18T14:15:44Z

hey @mrousavy i think we should keep this issue open, i'm facing the same issue since this function was released, can you provide a minimal example that's working for you?

rodgomesc · 2023-10-18T14:41:39Z

some weirdness scope things happening

UseCase1:

logcat throws JSI rethrowing JS exception: Regular javascript function cannot be shared. Try decorating the function with the 'worklet' keyword to allow the javascript function to be used as a worklet."

  const frameProcessor = useFrameProcessor(
    (frame) => {
      "worklet";
      runAsync(frame, () => {
        "worklet";
        console.log("my awesome log");
      });
    },
    []
  );

UseCase2:

do not throws anything onlogcat but frameContext is always undefined, the callback is not being triggered for some reason

  const myFn = (frameContext) => {
    "worklet";
    console.log("Here", frameContext);
  };

  const frameProcessor = useFrameProcessor(
    (frame) => {
      "worklet";
      runAsync(frame, myFn);
    },
    [myFn]
  );

mrousavy · 2023-10-18T15:00:26Z

Oh! Hm that's weird, I remember this worked when I built it.

@chrfalch do you maybe have any insights?

rodgomesc · 2023-10-18T15:01:38Z

oh wait

it seems that by design the frame it's not being passed to myFn as a callback prop, this forces us to uses arrow functions to hook the frame scope, also it seems that's no guarantees that internal.decrementRefCount() is being called after my long running function finishes the execution?, which causes a java.lang.IllegalStateException: Image is already closed,

react-native-vision-camera/package/src/FrameProcessorPlugins.ts

Lines 50 to 55 in 764897d

    
           runOnAsyncContext = Worklets.createRunInContextFn((frame: Frame, func: () => void) => { 
        
             'worklet' 
        
             try { 
        
               // Call long-running function 
        
               func() 
        
             } finally {

mrousavy · 2023-10-18T15:38:01Z

Wait what? not sure if I follow -

it seems that by design the frame it's not being passed to myFn as a callback prop

no you're right, you can just use the Frame inside your lambda directly, the lambda has no parameters.

there's no guarantees that internal.decrementRefCount() is being called after my long running function finishes the execution

Why not? It runs in finally, no?

rodgomesc · 2023-11-09T02:21:28Z

@mrousavy yep you are totally right, i was very drunk that day sorry

found a pattern that doesn't make sense for me, example app works my app doesn't work, tried different combination of libs and now i'm using same version that's on example but it always end up with

rodgomesc · 2023-11-09T02:26:00Z

i have a serie of frame processors at this moment running in sequence

something like this

const frameProcessor = (frame) => {
    'worklet';

     // processors
     const faces = faceDetectorProcessor(frame)
     const luminanceResults = luminanceDetectionProcessor(frame, faces)
     const spoofResults = antiSpoofingProcessor(frame);


    // callbacks
    handleSpoofingDetection(spoofResults)
    handleLuminanceDetection(luminanceResults)

},[handleSpoofingDetection, handleLuminanceDetection])

the problem is that at this point the frame are being deallocated before i finish my operations,
so i'm thinking to implement something like this, to drop some frames while my processors still running

const frameProcessor = (frame) => {
    'worklet';

    // just drop the frame
    if(!jsiUtil.finishedOperations) return;  

    const frameCopy = jsiUtil.copyFrame(frame);

     // processors      
     const faces = faceDetectorProcessor(frameCopy)
     const luminanceResults = luminanceDetectionProcessor(frameCopy, faces)
     const spoofResults = antiSpoofingProcessor(frameCopy);

    // cleanup frame copy
    jsiUtil.ReleaseFrameCopy()    

    // callbacks
    handleSpoofingDetection(spoofResults)
    handleLuminanceDetection(luminanceResults)

},[handleSpoofingDetection, handleLuminanceDetection])

would be nice to hear your opinion on this temporary solution to make sure i'm not missing anything

some final notes:

i'm linking my lib with visionCamera.so and unwraping the frame, i've tried increment refcount before my operations and decrement on the last one, but it seems to be ignored need further investigation

mrousavy · 2023-11-09T09:54:32Z

Woah, that's a weird error. Can you add console.log statements to your code (begin and end of sync calls, and begin and end of async calls) and then also to the native Frame's release method?

This is weird and you shouldn't need such workarounds - it should just work magically with ref counting.

rodgomesc · 2023-12-05T00:06:38Z

i still couldn't figure out what's happening, i'll try to upload something reproducible

rodgomesc · 2023-12-05T02:26:26Z

Haaa I found out, i have a monorepo with this structure

-- packages
---- playground
---- mylib

if i have reanimated in my lib i face this issue, removed reanimated from mylib and now i'm using it only in the playground, working like a charm

Edit1:

if i try a simple console.log after removing reanimated lib it works

useFrameProcessor(frame => .....

 runAsync(frame, () => {
 'worklet'
    console.log('hello from runAsync')
  })
  
},[])

if i try to use any function inside of it, it crashes exactly with same error as before

 const myFunc = () => {
    'worklet'
     console.log('hello from runAsync')
 }
 
useFrameProcessor(frame => .....

 runAsync(frame, () => {
  'worklet'
    myFunc();
  })
  
},[myFunc])

rodgomesc · 2023-12-05T18:03:56Z

@mrousavy fyi it's reproducible on example app now

mrousavy · 2023-12-05T19:48:57Z

Wait so Reanimated makes it break? Yea it's a bit weird that Worklets and reanimated do the same thing... Maybe we can fix this compatibility for now and think about a better solution in the future.

rodgomesc · 2023-12-06T18:08:49Z

Wait so Reanimated makes it break?

that's what i thought because after removing reanimated i can do a console.log inside the runAsync, however if i call any function inside it i got the same error

gbark · 2023-12-06T19:17:57Z

Similar issue here. Using @ismaelmoreiraa/vision-camera-ocr, any call to scanOCR using runAsync throws:

JSI rethrowing JS exception: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.facebook.jni.MapIteratorHelper" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/system_ext/lib64, /system/lib64, /system/system_ext/lib64]]

Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.facebook.jni.MapIteratorHelper" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/system_ext/lib64, /system/lib64, /system/system_ext/lib64]]
    at call (native)
    at scanOCR (/Users/username/dev/app/node_modules/@ismaelmoreiraa/vision-camera-ocr/src/index.tsx:8:21)
    at fn (native)
    at anonymous (/Users/username/dev/app/packages/app/src/features/foo.tsx:18:27)
    at fn (native)
    at anonymous (/Users/username/dev/app/node_modules/react-native-vision-camera/src/FrameProcessorPlugins.ts:6:9)

Running the same exact same code using runAtTargetFps works fine.

I'm not using Reanimated in my project.

mrousavy · 2024-01-11T16:19:07Z

Btw.; here's an explanation and a fix: margelo/react-native-worklets-core#136

mrousavy · 2024-01-15T12:57:40Z

This is not related to VisionCamera, but rather about Reanimated.

If you do not use Reanimated, runAsync works fine. If you do use Reanimated, you need to enable processNestedWorklets in the Reanimated's babel plugin. See software-mansion/react-native-reanimated#5576 for more info

Cosmorider · 2024-01-26T15:48:26Z

Enabling processNestedWorklets solved the error for me, but now this error pops up: Frame Processor Error: Value is undefined, expected an Object, js engine: VisionCamera

This is my code (from react-native-fast-tflite), note that the model is printed out, but then it shows the error, I'm trying to make the model not interfere with how the camera output is displayed in my app (it interferes on Android but not iOS):

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
  
    const data = resize(frame, {
      size: {
        // center-crop
        x: (frame.width / 2) - (320 / 2),
        y: (frame.height / 2) - (320 / 2),
        width: 320,
        height: 320,
      },
      pixelFormat: 'rgb',
      dataType: 'uint8'
    })

    runAsync(frame, () => {
      'worklet'
      console.log(model)

      const output = model.runSync([data])
    
      const numDetections = output[0]
    })
  }, [model])

If I pass the frame into runAsync, I get this error: ERROR Frame Processor Error: Exception in HostFunction: no ArrayBuffer attached, js engine: VisionCamera

Not sure if I'm misunderstanding some fundamentals here.

"react-native-vision-camera": "^3.8.2",
"react-native-worklets-core": "^0.2.4",
"react-native-reanimated": "^3.6.1",

mrousavy · 2024-01-26T15:51:51Z

ArrayBuffers can't be shared (right @chrfalch?), do the resize also in runAsynx.

chrfalch · 2024-01-26T15:52:42Z

Correct.

Cosmorider · 2024-01-26T16:06:44Z

Sorry, I meant that I passed the resize into runAsync and then I got this error, if that was what you were asking for: ERROR Frame Processor Error: Exception in HostFunction: no ArrayBuffer attached, js engine: VisionCamera

mrousavy · 2024-01-26T17:34:54Z

Yeah - try to move the resize call also into runAsync.

Cosmorider · 2024-01-26T18:03:18Z

I did that, and I get this error:
ERROR Frame Processor Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.mrousavy.camera.frameprocessor.Frame" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/product/lib64, /system/lib64, /system/product/lib64]], js engine: VisionCamera

When I use this code:

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'

    runAsync(frame, () => {
      'worklet'

      const data = resize(frame, {
        size: {
          // center-crop
          x: (frame.width / 2) - (320 / 2),
          y: (frame.height / 2) - (320 / 2),
          width: 320,
          height: 320,
        },
        pixelFormat: 'rgb',
        dataType: 'uint8'
      })
    })
  }, [])

Edit: If I have both resize outside and inside runAsync, it just crashes with no symbolic stack trace:

const frameProcessor = useFrameProcessor((frame) => {
    'worklet'

    const data = resize(frame, {
      size: {
        // center-crop
        x: (frame.width / 2) - (320 / 2),
        y: (frame.height / 2) - (320 / 2),
        width: 320,
        height: 320,
      },
      pixelFormat: 'rgb',
      dataType: 'uint8'
    })

    runAsync(frame, () => {
      'worklet'

      const data = resize(frame, {
        size: {
          // center-crop
          x: (frame.width / 2) - (320 / 2),
          y: (frame.height / 2) - (320 / 2),
          width: 320,
          height: 320,
        },
        pixelFormat: 'rgb',
        dataType: 'uint8'
      })
    })
  }, [])

rodgomesc · 2024-01-27T00:47:45Z

I did that, and I get this error: ERROR Frame Processor Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.mrousavy.camera.frameprocessor.Frame" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/product/lib64, /system/lib64, /system/product/lib64]], js engine: VisionCamera

that so weird, same problem here

mrousavy · 2024-01-29T10:56:03Z

From reading the error message, it looks like the reason the JNI class cannot be found in runAsync is because there is no JNI Environment set up on that specific Thread - which is weird, because we set it up here: https://github.com/margelo/react-native-worklets-core/blob/d8dae58ffac6b7050bb0b410b69acc621dcf74d8/cpp/WKTJsiWorkletContext.cpp#L147-L149

Is that not called when invoking a worklet from JS, @chrfalch ?

mrousavy · 2024-01-30T10:03:49Z

Hey @rodgomesc and @Cosmorider - a discord user posted a SIGABRT stacktrace with symbols attached so I was able to figure out that this came from JVisionCameraScheduler - can you guys maybe try if this PR fixes the issue for you? #2457

MSchmidt · 2024-01-30T12:34:56Z

I'm not seeing the same error. I have this one when using runAsync:

Frame Processor Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.facebook.jni.MapIteratorHelper" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system_ext/lib64, /system/lib64, /system_ext/lib64]], js engine: VisionCamera

The new PR does not fix this particular one.

mrousavy · 2024-01-30T13:04:11Z

@MSchmidt can you try to add a

facebook::jni::ThreadScope scope;

right at the top of this method:

react-native-vision-camera/package/android/src/main/cpp/frameprocessor/java-bindings/JFrameProcessor.cpp

Lines 35 to 45 in 02bc8a9

    
           void JFrameProcessor::callWithFrameHostObject(const std::shared_ptr<FrameHostObject>& frameHostObject) const { 
        
             // Call the Frame Processor on the Worklet Runtime 
        
             jsi::Runtime& runtime = _workletContext->getWorkletRuntime(); 
        
             // Wrap HostObject as JSI Value 
        
             auto argument = jsi::Object::createFromHostObject(runtime, frameHostObject); 
        
             jsi::Value jsValue(std::move(argument)); 
        
             // Call the Worklet with the Frame JS Host Object as an argument 
        
             _workletInvoker->call(runtime, jsi::Value::undefined(), &jsValue, 1); 
        
           }

? Let me know if that works for you

mrousavy · 2024-01-30T13:05:03Z

@MSchmidt in any way to find a fix myself I would need you to also get me a symbolicated stacktrace/crash log just like the one I have posted in PR #2457.

mrousavy · 2024-01-30T13:06:47Z

A second idea would be that I am using JMap's iterator function here:

react-native-vision-camera/package/android/src/main/cpp/frameprocessor/JSIJNIConversion.cpp

Lines 155 to 166 in 02bc8a9

    
           // Map<K, V> 
        
           auto map = static_ref_cast<JMap<jstring, jobject>>(object); 
        
           auto result = jsi::Object(runtime); 
        
           for (const auto& entry : *map) { 
        
             auto key = entry.first->toString(); 
        
             auto value = entry.second; 
        
             auto jsiValue = convertJNIObjectToJSIValue(runtime, value); 
        
             result.setProperty(runtime, key.c_str(), jsiValue); 
        
           } 
        
           return result;

..maybe you are using an older version of fbjni where the map iterator part doesn't exist yet, hence it cannot find com.facebook.jni.MapIteratorHelper?

MSchmidt · 2024-01-30T13:21:41Z

facebook::jni::ThreadScope scope; added to package/android/src/main/cpp/frameprocessor/java-bindings/JFrameProcessor.cpp is not working.

It's the same as this one: #1791 (comment)

It's somehow related to reanimated still. Their plugin does something different from vision camera's. It works when reanimated is thrown out of the project. Which isn't an option though.

mrousavy · 2024-01-30T14:32:34Z

It's the same as this one: #1791 (comment)

Yea I still need a symbolicated stacktrace. The error linked above just tells me that something inside the Frame Processor crashed, but I need the native C++ stacktrace.

Also the guy (@gbark) said he is not using Reanimated in his project, which makes me doubt that this is something coming from reanimated.

mrousavy · 2024-01-30T15:08:36Z

Okay nevermind - I was able to reproduce this 💪 - this fixes the issue: margelo/react-native-worklets-core#141 🎉

mrousavy · 2024-01-30T15:14:52Z

Fixed in react-native-worklets-core 0.3.0 🎉🥳

namnotfake · 2024-04-29T10:45:34Z

This is not related to VisionCamera, but rather about Reanimated.

If you do not use Reanimated, runAsync works fine. If you do use Reanimated, you need to enable processNestedWorklets in the Reanimated's babel plugin. See software-mansion/react-native-reanimated#5576 for more info

I tried to fix it but it didn't work

manolo-battista added the 🐛 bug Something isn't working label Sep 11, 2023

mrousavy closed this as completed Sep 30, 2023

mrousavy reopened this Nov 9, 2023

mrousavy closed this as completed Jan 15, 2024

mrousavy reopened this Jan 29, 2024

mrousavy mentioned this issue Jan 30, 2024

fix: Fix Unable to retrieve jni environment. Is the thread attached? errors by using jni::ThreadScope #2457

Merged

mrousavy closed this as completed in #2457 Jan 30, 2024

mrousavy mentioned this issue Jan 30, 2024

fix: Use ThreadScope::WithClassLoader to access custom JNI fields margelo/react-native-worklets-core#141

Merged

This was referenced Feb 14, 2024

🐛 "no non-static method" error only on release mode #2544

Closed

IOS: example not work as expected luicfrr/react-native-vision-camera-face-detector#5

Closed

🐛 V3 runAsync doesn't work #1791

🐛 V3 runAsync doesn't work #1791

Comments

manolo-battista commented Sep 11, 2023 • edited

What's happening?

Reproduceable Code

Relevant log output

Camera Device

Device

VisionCamera Version

Can you reproduce this issue in the VisionCamera Example app?

Additional information

abdelrahman-muntaser commented Sep 21, 2023

manolo-battista commented Sep 21, 2023

mrousavy commented Sep 30, 2023

rodgomesc commented Oct 18, 2023

rodgomesc commented Oct 18, 2023 • edited

UseCase1:

UseCase2:

mrousavy commented Oct 18, 2023

rodgomesc commented Oct 18, 2023 • edited

mrousavy commented Oct 18, 2023

rodgomesc commented Nov 9, 2023

rodgomesc commented Nov 9, 2023 • edited

mrousavy commented Nov 9, 2023

rodgomesc commented Dec 5, 2023

rodgomesc commented Dec 5, 2023 • edited

rodgomesc commented Dec 5, 2023

mrousavy commented Dec 5, 2023

rodgomesc commented Dec 6, 2023 • edited

gbark commented Dec 6, 2023 • edited

mrousavy commented Jan 11, 2024

mrousavy commented Jan 15, 2024

Cosmorider commented Jan 26, 2024

mrousavy commented Jan 26, 2024

chrfalch commented Jan 26, 2024

Cosmorider commented Jan 26, 2024

mrousavy commented Jan 26, 2024

Cosmorider commented Jan 26, 2024 • edited

rodgomesc commented Jan 27, 2024

mrousavy commented Jan 29, 2024

mrousavy commented Jan 30, 2024

MSchmidt commented Jan 30, 2024

mrousavy commented Jan 30, 2024

mrousavy commented Jan 30, 2024

mrousavy commented Jan 30, 2024

MSchmidt commented Jan 30, 2024

mrousavy commented Jan 30, 2024

mrousavy commented Jan 30, 2024

mrousavy commented Jan 30, 2024

namnotfake commented Apr 29, 2024

manolo-battista commented Sep 11, 2023 •

edited

rodgomesc commented Oct 18, 2023 •

edited

rodgomesc commented Oct 18, 2023 •

edited

rodgomesc commented Nov 9, 2023 •

edited

rodgomesc commented Dec 5, 2023 •

edited

rodgomesc commented Dec 6, 2023 •

edited

gbark commented Dec 6, 2023 •

edited

Cosmorider commented Jan 26, 2024 •

edited