Support React Native

[hans00]

Make it support React Native.

And I made a example for it.
https://github.com/hans00/react-native-transformers-example

On Android need some library patch.

TODO:

• Check models are works fine
• Research more efficiently image processing

[xenova]

Woah nice! There are a lot of people who are interested in doing stuff like this! Looking forward to reviewing this when you're ready! cc @pcuenca

[jhen0409]

I'm investigating a performance issue with Whisper tiny.en, and it looks like the performance is not as expected in the example.

I quickly add log for onnxruntime-react-native/lib/backend.ts, this is the result:

# Encode result
 LOG  ONNX session run finished 767 ms

# Decode result
 LOG  ONNX session run finished 4518 ms
# ... 4 runs ...
 LOG  ONNX session run finished 4856 ms
 LOG  Time: 40016
 LOG  Result: {"text": " (buzzing)"}

Click to expand

const t0 = performance.now()
// this.#inferenceSession === NativeModules.Onnxruntime
const results: Binding.ReturnType = await this.#inferenceSession.run(this.#key, input, outputNames, options);
const output = this.decodeReturnType(results);
console.log('ONNX session run finished', performance.now() - t0)

Also see logs of native part (added log to OnnxruntimeModule.java), it is significantly less than the time taken by JS part:

# Encode result
09:47:59.203 ONNXRUNTIME_RN run() is finished, time: 273 ms

# Decode result
09:48:00.280 ONNXRUNTIME_RN run() is finished, time: 339 ms
# ... 4 runs ...
09:48:23.807 ONNXRUNTIME_RN run() is finished, time: 541 ms

I think this problem may come from blocking caused by native module passing too large data. I think request/help onnxruntime for migrate to JSI module may solve this problem.

EDIT: It seems the logger have some bugs leads me to think that the problem is from the native bridge. I add timeout await to the decodeReturnType call and found the issue is from this function.

[Sly777]

@xenova is there any timeline for checking and merging this?

[jpcoder2]

There any update on this? This is a great javascript library and I've built a nice prototype that runs (barely) in mobile Safari but I would much rather have it running in a React Native app. I've tried the beta library from @hans00 but could never really get it to work. Would be great if this library had official React Native support with working examples on iOS and Android.

Thanks for all you do...

[hans00]

There any update on this? This is a great javascript library and I've built a nice prototype that runs (barely) in mobile Safari but I would much rather have it running in a React Native app. I've tried the beta library from @hans00 but could never really get it to work. Would be great if this library had official React Native support with working examples on iOS and Android.

Thanks for all you do...

Here is my example, it should work with the latest beta although the last update was 4 months ago.
https://github.com/hans00/react-native-transformers-example

And here is a simple guide to make transformers work on React Native.
https://github.com/hans00/react-native-transformers-example/blob/main/DEVELOPMENT.md

[jpcoder2]

Thanks for the reply @hans00. I tried your example app but could never get it to build.

I tried using your guide with a new React Native+Expo app and I'm getting this error. This happens at this line:

let pipe = await pipeline('sentiment-analysis');

Any idea how to fix?

Thanks for your help!

[jpcoder2]

@hans00, I'm thinking that the NativeEventsEmitter error is because of one of the dependencies that is not compatible with the current version of Expo. I'm not sure which one though and expo install --fix didn't fix the issue.

There any way you could create a new sample app that uses the current version of React Native+Expo?

[hans00]

@hans00, I'm thinking that the NativeEventsEmitter error is because of one of the dependencies that is not compatible with the current version of Expo. I'm not sure which one though and expo install --fix didn't fix the issue.

It seems react-native-fs not linked in your dev app.
https://github.com/itinance/react-native-fs/blob/64aa755cc1d37f59fa205bf2d52dd71a7d691504/FS.common.js#L10-L14

[jpcoder2]

@hans00, after I run "npm link react-native-fs" and then run again I get this.

[hans00]

@hans00, after I run "npm link react-native-fs" and then run again I get this.

Not sure why your project not auto map lib.

It should be auto mapped by "react-native" field
https://github.com/hans00/transformers.js/blob/9f30f1b7feedad578dc93bed4c24db9cf8febd62/package.json#L79-L86

Alternatively you could manually map using babel-plugin-module-resolver

[jpcoder2]

@hans00, for whatever reason it is not mapped. Maybe something to do with Expo?

Do you have an example of how to map this with babel-plugin-module-resolver?

[jpcoder2]

@hans00, well, I may have gotten past the fs mapping problem, but after installing babel-plugin-module-resolver and using this .babelrc I get the below error.

{
"plugins": [
["module-resolver", {
"alias": {
"fs": "react-native-fs"
}
}]
]
}

[simonwh]

👍 +1 for getting this merged - anything we can do? @xenova

[hans00]

👍 +1 for getting this merged - anything we can do? @xenova

Currently v3 priority is higher than this PR.

[hans00]

@hans00, well, I may have gotten past the fs mapping problem, but after installing babel-plugin-module-resolver and using this .babelrc I get the below error.

{ "plugins": [ ["module-resolver", { "alias": { "fs": "react-native-fs" } }] ] }

Hmm... Maybe you can look others try.

#118 (comment)
#118 (comment)

[jpcoder2]

@hans00, I found another way to use transformers.js in React Native that works with Expo so I'll just do that until the official React Native support is more stable.

Thanks for your help though.

[TowhidKashem]

@hans00, I found another way to use transformers.js in React Native that works with Expo so I'll just do that until the official React Native support is more stable.

Thanks for your help though.

hey @jpcoder2 mind sharing how?

[jpcoder2]

hey @jpcoder2 mind sharing how?

Yea, I wound up creating a static web server using @dr.pogodin/react-native-static-server and running a hidden WebView with a little static page loaded from that local web server with some javascript in it to allow posting messages back in forth between RN and the browser to talk to a transformer.js running in that hidden browser. Not pretty, but it seems to work pretty well, at least on iOS. I haven't tried it on Android yet. Obviously a native Expo solution will be much more ideal but this works for now.

[jpcoder2]

@hans00 I noticed you synced up your merge. Is this getting closer to getting approved? And if so, will there be an official RN/Expo example when it's merged in? Thanks for all this!

[TechSaq]

@hans00, I found another way to use transformers.js in React Native that works with Expo so I'll just do that until the official React Native support is more stable.

Thanks for your help though.

Can you please tell how you integrated in Expo?