Expose native worklet API

[mrousavy]

Description

This PR exposes the native "worklet" API by lifting the "manage-a-runtime" logic out of NativeReanimatedModule so that other native modules can implement this for their own use case. This "manage-a-runtime" logic is implementable by the RuntimeManager base class.

Advantages

• Other native libraries can make use of Reanimated's workletization functionality to spawn new JS threads. Example: My camera library: react-native-vision-camera: Creating "frame processors" to run QR Code scanning, AI, Realtime Video Chats, etc
• Splitting the SharedValue parts from the NativeReanimatedModule, making everything a bit more "decoupled"

Related

• Makes use of a few parts of Multithreading with worklets #1561
• Is required for my frame processors: JSI Frame Processors (iOS) mrousavy/react-native-vision-camera#2

Approach

I will be starting from this location in my VisionCamera project, I am trying to create a worklet here.

"Creating a worklet" = Copy the function from the default React JS Runtime to the newly created one for the frame processors, and copy over all variables from "outside".

So ShareableValue::adapt is my starting point, from there on I will start to remove all NativeReanimatedModule references and will try to decouple it as much as possible.

Solution

The current solution works fine and nothing has changed for Reanimated. No side-effects whatsoever (afaik).

But I am not 100% happy with one thing: RuntimeManager has a Scheduler, ValueSetter and ErrorHandler which are all properties that are really specific to Reanimated's NativeReanimatedModule. Ideally other native modules shouldn't be using those 3 things, since ValueSetter is specific for mirroring styles from native to JS (afaik), ErrorHandler just displays the errors in worklets (afaik) and scheduler is responsible for dispatching from Worklet -> JS and vice versa. Ideally I want to also abstract those away with the following solution:

• errorHandler is never needed, the default React LogBox should always be used for any errors in worklets and should even display source frames. For any other calls, just throw an std::exception (or jsi::JSError)
• valueSetter is not needed at all, this is only needed for Reanimated. Would be better to remove that from RuntimeManager too.
• scheduler is needed to use runOnJS I believe, no idea if we really need that. It's also really hard to create an instance of the Scheduler on Android...

@Szymon20000 I would like your opinion on those things. I guess we can merge this PR since no side effects are introduced and it's now possible for other libraries to hook into this logic and concentrate on those 3 things in another PR. Ideally it should be really simple to create worklets, but now it requires you to create an ErrorHandler and Scheduler.

Edit

As for the scheduler, I haven't tried it yet but what happens if I do the followng:

const sv = useSharedValue(0)
const frameProcessor = useFrameProcessor(() => {
  'worklet'
  sv.value = 5
}, [])
// the above frameProcessor is created in my custom jsi::Runtime and workletized with ShareableValue::adapt

sv.value = 5 will be executed in my custom jsi::Runtime from the Frame Processor, so it will not be updated on the UI thread. Would this still work? Is this like updating the value on the JS thread?

[mrousavy]

cc @Szymon20000 @karol-bisztyga @piaskowyk

[mrousavy]

it compiles 🎉

[mrousavy]

I've refactored the code so that other libraries can inherit RuntimeManager.h which provides the following properties:

• runtime: An individual runtime, effectively a secondary JS thread/process, similar to web workers - or how REA does it for the UI thread
• valueSetter: A ShareableValue (workletized JS function) which is responsible for setting a value in JS (afaik used to reflect native changes in the useShareableValue hook, correct me if I'm wrong) (nullable!) I'm not sure if we even need to provide this for other libraries, since it's pretty REA specific, let me know what you think @ software mansion
• errorHandler: Handles errors. Same thing as with valueSetter, not sure if that should be provided - if not, we have to refactor a bit more of the code that calls the RuntimeManager's errorHandler (let me know what you think @ software mansion)
• scheduler: Schedules code on the UI thread using platform specific code, or on the JS thread using the CallInvoker from React. Same as above, not sure if we need to provide that. let me know what you think @ software mansion

those 4 properties are just extracted from NativeReanimatedModule.h

While everything compiles correctly (and uses the same semantics, so everything REA related works as expected), I'm not sure what's missing here because when I call ShareableValue::adapt in my VisionCamera (see this line), it still gives me a "BAD_ACCESS" error and crashes the app. It does successfully run through the big if (these lines here), but after a few runs it crashes. I am not exactly sure where and when, but I noticed that it at least gets to this line.

I'd appreciate any feedback from you guys, maybe you can spot an error in my code - or let me know if there's any additional work required for this to work.

cc @Szymon20000 @karol-bisztyga @piaskowyk @terrysahaidak @kmagiera

[mrousavy]

Managed to get it working successfully:

Screen.Recording.2021-03-11.at.10.54.51.mov

This means, that this PR is technically ready. I'd like you guys to answer my questions in the comment above on how we're going to proceed with this in terms of code organisation (regarding the valueSetter errorHandler and scheduler), but everything works now. 🔥

[mrousavy]

As a side note, I noticed that you guys did a lot of

std::shared_value<T> someValue(new T(...));

instead of

std::shared_value<T> someValue = std::make_shared<T>(...);

is there a specific reason for that?

Might be a micro-optimization, but make_shared is faster than new as it only has to allocate memory once (and new has to do 1x for T, 1x for control-block)

EDIT: Created a separate PR to replace all new calls: #1820

[mrousavy]

I made a multithreading library by using the changes in this PR: https://github.com/mrousavy/react-native-multithreading

[mrousavy]

I noticed that I can't simply link the Android Reanimated native code against my library because it is pre-built using an .aar, so this whole "Expose native worklet API" only works on iOS atm.

Since react-native is starting to use more and more C++ code (Rearchitecture, TurboModules), and with the release of RN 0.64 even requires the user to install the NDK (see template's build.gradle), I think it wouldn't be too far off to not use a prebuilt .aar library for reanimated, but instead ship the .h/.cpp sources and let the user compile it themselves. (That is no extra effort for the user, Android/Gradle manages all of that automatically.)
This would allow my VisionCamera, my Multithreading library and potentially other libraries to use native Reanimated code to allow "workletization" - also it's way easier to patch something in the lib. Plus it requires less effort for you to release a new npm version. What do you guys think?

[mrousavy]

Closing in favor of #1861

[Szymon20000]

We should add a possibility to pass additional hooks/functions as options in babel.config.js. https://babeljs.io/docs/en/plugins/#plugin-options
It cannot be directly in the plugin code as it's not part of reanimated.