-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
馃挱 Two separate frame processors at different target FPS #2891
Comments
why not just const frameProcessor = useFrameProcessor((frame) => {
'worklet'
runAtTargetFps(15, () => {
lowHzFrameProcessor(frame)
})
// optional; either `runAsync`, or just call directly. whatever works for you
runAsync(frame, () => {
'worklet'
highHzFrameProcessor(frame)
})
}) ? |
馃う Thanks! I think a combination of working late on a Sunday + brainfart made me not realize I could do that. I think I just misunderstood how the runAtTargetFps function worked. Appreciate the answer! |
haha no worries - glad it helped! |
Sorry to bring this up again, but I'm still occasionally running into issues where the image is closed too early. Here's my current implementation: // what I'm doing in RN
const frameProcessor = useFrameProcessor((frame: Frame) => {
'worklet'
runAtTargetFps(5, () => {
runAsync(frame, () => {
'worklet'
runSlowPlugin(frame);
});
});
runFastPlugin(frame);
}); // my native plugins
// slow plugin implementation
override fun callback(frame: Frame, params: MutableMap<String, Any>?): Boolean {
val image = frame.image ?: return false
// should be nearly instant as I'm just moving data around
// However, I think it may be sometimes slower than the
// fast plugin which is why I think there is the
// "image is already closed" issue
val buffers = extractBuffers(image)
runVeryLongProcess(image)
return false
}
// fast plugin implementation
override fun callback(frame: Frame, params: MutableMap<String, Any>?): Double {
val image = frame.image ?: return 0.0
// do some processing
val value = // ...
// I've been closing it in this plugin because it's
// the plugin that always runs and for the most part
// it seems to work fine
image.close()
return val
} I've seen the information about writing an asynchronous frame copy, but I'm not sure how to do it in kotlin. The documentation seems to indicate that I can do something like: val frameCopy = Frame(
frame.image,
frame.timestamp,
frame.orientation,
frame.isMirrored
)
runVeryLongProcess(frameCopy.image) But I'm still occasionally getting the "image is closed" issue. A temporary fix I had was to add a parameter to the fast plugin to prevent it from closing the image and then always closing it in the slow plugin. However, if the slow one happened to run very quickly then it would throw the error, and I had to keep track of the frame count to determine if I should try running the 5 fps frame plugin. Is there any documentation on how to copy the image in kotlin without it being closed? Alternatively, is there a way to detect if the image is already closed and just return early instead of trying to continue the plugins? If I don't ever explicitly close the image will I have memory leaks or is Android smart enough to garbage collect it? Feel free to close this again. I didn't want to open a new issue without the context but I understand it's a little disorganized if threads keep reopening. |
Why? Why are you closing the Frame? The documentation tells you that you should not close the Frame, VisionCamera handles that for you. |
Oh, okay. Thanks. I couldn't find documentation about closing the frame on the RNVC website, do you know which section it's in? I tried searching for "close" on the website and also in the /**
* Closes and disposes the Frame.
* Only close frames that you have created yourself, e.g. by copying the frame you receive in a frame processor.
*
* @example
* ```ts
* const frameProcessor = useFrameProcessor((frame) => {
* const smallerCopy = resize(frame, 480, 270)
* // run AI ...
* smallerCopy.close()
* // don't close `frame`!
* })
* ```
*/
close(): void;
} If you get a chance could you take a look and see if it's still in the documentation? Definitely not a priority but if you can't find it either I could make a PR adding it to "Creating Frame Processor Plugins (Android)" section of the docs website, just let me know! Thanks again. Your super quick responses make working with RNVC one of the best experiences I've had in the couple years I've been working in mobile development. |
Again, you should not close the Frame. If you intend to keep it longer than a runAsync call (eg when it goes out of scope in your native plugin) call incrementRefCount() first, and then decrementRefCount() after you're done. |
Question
I have two separate native frame processors that I would like to run. However, I only want to run one of the frame processors at 15 hz (it's async and I don't need high temporal resolution), and the other one at 60-120 hz (not async, need the high temporal resolution but I've already optimized it so that it can run at 60-120 hz).
What I tried
Currently, I've been keeping track of the total frame count in a shared value and just running the other frame processor every couple frames. I was wondering if there was a more optimal way to do it such as using the runAtTargetFps() in separate worklets.
This is my current implementation:
Note: on Android, I'm not calling
frame.image.close()
in the low frequency frame processor so that the image doesn't get garbage collected before the high frequency processor runs.VisionCamera Version
3.9.2
Additional information
The text was updated successfully, but these errors were encountered: