feat: Sync Frame Processors #1472

mrousavy · 2023-02-13T14:29:52Z

What

Before, Frame Processors ran on a separate Thread.

After, Frame Processors run fully synchronous and always at the same FPS as the Camera.

Two new functions have been introduced:

runAtTargetFps(fps: number, func: () => void): Runs the given code as often as the given fps, effectively throttling it's calls.
runAsync(frame: Frame, func: () => void): Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute.

You can use runAtTargetFps to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use runAtTargetFps(25, ...).)

You can use runAsync to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side.

You can also combine both functions.

Examples:

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")
}, [])

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAtTargetFps(10, () => {
    'worklet'
    console.log("I'm running at 10 FPS!")
  })
}, [])

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAsync(frame, () => {
    'worklet'
    console.log("I'm running on another Thread, I can block for longer!")
  })
}, [])

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running at 60 FPS!")

  runAtTargetFps(10, () => {
    'worklet'
    runAsync(frame, () => {
      'worklet'
      console.log("I'm running on another Thread at 10 FPS, I can block for longer!")
    })
  })
}, [])

Changes

Tested on

Related issues

github-actions · 2023-02-13T16:40:20Z

`yarn.lock` changes

Click to toggle table visibility

Name	Previous	Current
`jsc-android`	250230.2.1	250231.0.0
`react-native-codegen`	0.71.3	0.71.5
`react-native-gradle-plugin`	0.71.14	0.71.15
`react-native-worklets`	-	0.1.0
`react-native`	0.71.2	0.71.3

github-actions · 2023-02-13T16:40:46Z

src/FrameProcessorPlugins.ts

+ * })
+ * ```
+ */
+export function runAsync(func: () => void): void {


[tsc] <6133> _{reported by reviewdog 🐶}
'func' is declared but its value is never read.

github-actions · 2023-02-13T16:41:39Z

src/FrameProcessorPlugins.ts

@@ -11,3 +12,77 @@ type TFrameProcessorPlugins = Record<string, FrameProcessor>;
 * All natively installed Frame Processor Plugins.
 */
 export const FrameProcessorPlugins = global.FrameProcessorPlugins as TFrameProcessorPlugins;


[tsc] <7017> _{reported by reviewdog 🐶}
Element implicitly has an 'any' type because type 'typeof globalThis' has no index signature.

github-actions · 2023-02-13T17:10:07Z

android/src/main/cpp/FrameHostObject.cpp

@@ -80,10 +81,26 @@ jsi::Value FrameHostObject::get(jsi::Runtime& runtime, const jsi::PropNameID& pr
    this->assertIsFrameStrong(runtime, name);
    return jsi::Value(this->frame->getPlanesCount());
  }
+  if (name == "refCount") {
+    return jsi::Value((double) _refCount);


[cpplint] _{reported by reviewdog 🐶}
Using C-style cast. Use static_cast(...) instead [readability/casting] [4]

github-actions · 2023-02-13T17:10:07Z

android/src/main/cpp/FrameHostObject.cpp

+  auto name = propName.utf8(runtime);
+
+  if (name == "refCount") {
+    _refCount = (size_t) value.asNumber();


[cpplint] _{reported by reviewdog 🐶}
Using C-style cast. Use static_cast<size_t>(...) instead [readability/casting] [4]

github-actions · 2023-02-14T18:32:59Z

src/FrameProcessorPlugins.ts

+}
+
+runAtTargetFps.__initData = {
+  code: runAtTargetFps.asString,


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'asString' does not exist on type 'typeof runAtTargetFps'.

github-actions · 2023-02-14T18:33:00Z

src/FrameProcessorPlugins.ts

+
+runAtTargetFps.__initData = {
+  code: runAtTargetFps.asString,
+  location: runAtTargetFps.__location,


[tsc] <2339> _{reported by reviewdog 🐶}
Property '__location' does not exist on type 'typeof runAtTargetFps'.

github-actions · 2023-02-14T18:33:00Z

src/FrameProcessorPlugins.ts

+  'worklet';
+  // Increment ref count by one
+  frame.refCount++;
+  func.__initData = {


[tsc] <2339> _{reported by reviewdog 🐶}
Property '__initData' does not exist on type '() => void'.

github-actions · 2023-02-14T18:33:00Z

src/FrameProcessorPlugins.ts

+  // Increment ref count by one
+  frame.refCount++;
+  func.__initData = {
+    code: func.asString,


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'asString' does not exist on type '() => void'.

github-actions · 2023-02-14T18:33:00Z

src/FrameProcessorPlugins.ts

+  frame.refCount++;
+  func.__initData = {
+    code: func.asString,
+    location: func.__location,


[tsc] <2339> _{reported by reviewdog 🐶}
Property '__location' does not exist on type '() => void'.

github-actions · 2023-02-14T18:33:01Z

src/FrameProcessorPlugins.ts

+  };
+
+  const fn = Worklets.createRunInContextFn(wrapper, context);
+  fn.__initData = {


[tsc] <2339> _{reported by reviewdog 🐶}
Property '__initData' does not exist on type '() => Promise'.

github-actions · 2023-02-14T18:33:01Z

src/FrameProcessorPlugins.ts

+
+  const fn = Worklets.createRunInContextFn(wrapper, context);
+  fn.__initData = {
+    code: fn.asString,


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'asString' does not exist on type '() => Promise'.

github-actions · 2023-02-14T18:33:01Z

src/FrameProcessorPlugins.ts

+  const fn = Worklets.createRunInContextFn(wrapper, context);
+  fn.__initData = {
+    code: fn.asString,
+    location: fn.__location,


[tsc] <2339> _{reported by reviewdog 🐶}
Property '__location' does not exist on type '() => Promise'.

github-actions · 2023-02-14T18:33:01Z

src/FrameProcessorPlugins.ts

+}
+
+runAsync.__initData = {
+  code: runAsync.asString,


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'asString' does not exist on type 'typeof runAsync'.

github-actions · 2023-02-14T18:33:01Z

src/FrameProcessorPlugins.ts

+
+runAsync.__initData = {
+  code: runAsync.asString,
+  location: runAsync.__location,


[tsc] <2339> _{reported by reviewdog 🐶}
Property '__location' does not exist on type 'typeof runAsync'.

github-actions · 2023-02-15T12:40:09Z

android/src/main/cpp/FrameHostObject.cpp

@@ -80,10 +83,30 @@ jsi::Value FrameHostObject::get(jsi::Runtime& runtime, const jsi::PropNameID& pr
    this->assertIsFrameStrong(runtime, name);
    return jsi::Value(this->frame->getPlanesCount());
  }
+  if (name == "refCount") {
+    if (!_refCount) {
+      _refCount = std::make_shared<RNWorklet::JsiSharedValue>(jsi::Value(0),


[cpplint] _{reported by reviewdog 🐶}
Add #include for make_shared<> [build/include_what_you_use] [4]

github-actions · 2023-02-15T12:40:09Z

android/src/main/cpp/FrameHostObject.h

@@ -34,6 +37,7 @@ class JSI_EXPORT FrameHostObject : public jsi::HostObject {
  static auto constexpr TAG = "VisionCamera";

  void assertIsFrameStrong(jsi::Runtime& runtime, const std::string& accessedPropName) const; // NOLINT(runtime/references)
+  std::shared_ptr<RNWorklet::JsiSharedValue> _refCount;


[cpplint] _{reported by reviewdog 🐶}
Add #include for shared_ptr<> [build/include_what_you_use] [4]

github-actions · 2023-02-15T15:26:53Z

src/hooks/useFrameProcessor.ts

+    frameProcessor(frame);
+
+    // Potentially delete Frame if we were the last ref (no runAsync)
+    frame.refCount.value--;


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'refCount' does not exist on type 'Frame'.

github-actions · 2023-02-15T15:26:54Z

src/hooks/useFrameProcessor.ts

+
+    // Potentially delete Frame if we were the last ref (no runAsync)
+    frame.refCount.value--;
+    if (frame.refCount.value <= 0) frame.close();


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'refCount' does not exist on type 'Frame'.

github-actions · 2023-02-15T15:26:54Z

src/hooks/useFrameProcessor.ts

+
+    // Potentially delete Frame if we were the last ref (no runAsync)
+    frame.refCount.value--;
+    if (frame.refCount.value <= 0) frame.close();


[tsc] <2339> _{reported by reviewdog 🐶}
Property 'close' does not exist on type 'Frame'.

…1472) Before, Frame Processors ran on a separate Thread. After, Frame Processors run fully synchronous and always at the same FPS as the Camera. Two new functions have been introduced: * `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls. * `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute. You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.) You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side. You can also combine both functions. Examples: ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAtTargetFps(10, () => { 'worklet' console.log("I'm running at 10 FPS!") }) }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAsync(frame, () => { 'worklet' console.log("I'm running on another Thread, I can block for longer!") }) }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAtTargetFps(10, () => { 'worklet' runAsync(frame, () => { 'worklet' console.log("I'm running on another Thread at 10 FPS, I can block for longer!") }) }) }, []) ```

mrousavy added 8 commits February 13, 2023 15:27

Remove onFrameProcessorPerformanceSuggestionAvailable

9625b93

Remove Frame Processor FPS

0089f39

feat: Create runAtTargetFps

3089ba6

feat: Create runAsync template

6596e13

Add RN Worklets as a devDepenendy

02308c8

Add refCount to FrameHostObject

1329624

Simplify code a bit - no errors expected here

eb7b065

Close Frame in JS

588188c

mrousavy added 3 commits February 13, 2023 17:40

Make FP sync

241c784

Use SV for FP state

6fb8ecc

Add runAsync base

91d9869

github-actions bot reviewed Feb 13, 2023

View reviewed changes

mrousavy added 3 commits February 13, 2023 17:52

Make runAsync actually run on a background thread

f781c6b

fix: Fix runAtTargetFps implementation

3006dfc

Android add refCount

a038617

github-actions bot reviewed Feb 13, 2023

View reviewed changes

mrousavy added 7 commits February 13, 2023 18:31

Update RN Worklets

5f18f8e

Use JS based implementation for runAsync

42ebf51

Update package.json

06c0646

Use JSI_HOST_FUNCTION_LAMBDA

566d122

fix: Wait for func exec to decrement ref count

2facf7d

Update RN Worklets

41a7d28

Upgrade RN Worklets

10bdebf

github-actions bot reviewed Feb 14, 2023

View reviewed changes

mrousavy added 4 commits February 15, 2023 12:32

Update RN Worklets

65579e4

Update yarn.lock

508763f

Update FrameProcessorRuntimeManager.cpp

6603b04

Install REA from master

db64880

mrousavy added 2 commits February 15, 2023 13:37

Upgrade RN

2c1ff88

Make refCount a SharedValue

8894833

github-actions bot reviewed Feb 15, 2023

View reviewed changes

mrousavy added 7 commits February 15, 2023 13:45

fix: Update FP Plugin parameter types

f29a4ec

Use RefCount SV

b35fc3c

Upgrade RN Worklets

6a2739e

fix: Close Frame on other Context

98095e3

Remove logs

4660180

feat: Implement increment and decrement ref count on iOS

40a72c0

Update useFrameProcessor.ts

6bb5725

github-actions bot reviewed Feb 15, 2023

View reviewed changes

feat: Implement Ref Count on Android

a63da8b

mrousavy marked this pull request as ready for review February 15, 2023 15:46

mrousavy merged commit 30b5615 into v3 Feb 15, 2023

mrousavy deleted the feat/sync-frame-processors branch February 15, 2023 15:47

wuguishifu mentioned this pull request May 27, 2024

💭 Two separate frame processors at different target FPS #2891

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Sync Frame Processors #1472

feat: Sync Frame Processors #1472

mrousavy commented Feb 13, 2023 •

edited

Loading

github-actions bot commented Feb 13, 2023 •

edited

Loading

github-actions bot Feb 13, 2023

github-actions bot Feb 13, 2023

github-actions bot Feb 13, 2023

github-actions bot Feb 13, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 14, 2023

github-actions bot Feb 15, 2023

github-actions bot Feb 15, 2023

github-actions bot Feb 15, 2023

github-actions bot Feb 15, 2023

github-actions bot Feb 15, 2023

feat: Sync Frame Processors #1472

feat: Sync Frame Processors #1472

Conversation

mrousavy commented Feb 13, 2023 • edited Loading

What

Changes

Tested on

Related issues

github-actions bot commented Feb 13, 2023 • edited Loading

yarn.lock changes

github-actions bot Feb 13, 2023

Choose a reason for hiding this comment

github-actions bot Feb 13, 2023

Choose a reason for hiding this comment

github-actions bot Feb 13, 2023

Choose a reason for hiding this comment

github-actions bot Feb 13, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 14, 2023

Choose a reason for hiding this comment

github-actions bot Feb 15, 2023

Choose a reason for hiding this comment

github-actions bot Feb 15, 2023

Choose a reason for hiding this comment

github-actions bot Feb 15, 2023

Choose a reason for hiding this comment

github-actions bot Feb 15, 2023

Choose a reason for hiding this comment

github-actions bot Feb 15, 2023

Choose a reason for hiding this comment

mrousavy commented Feb 13, 2023 •

edited

Loading

github-actions bot commented Feb 13, 2023 •

edited

Loading

`yarn.lock` changes