react-native-vision-camera

[ciaoamigoschat]

Hello, I wanted to ask if your project supports react-native-vision-camera or if there's anyone who can guide me on how to configure it with react-native-webrtc.

[saghul]

You cannot use them together. WebRTC opens the camera directly.

[ciaoamigoschat]

I don't want to use them together, I want to use react-native-vision-camera instead of the default one.

[saghul]

So you do want to use them together then, You cannot use vision camera as a source for WebRTC because it opens the camera directly.

[ciaoamigoschat]

I found this information on the react-native-vision-camera docs: https://react-native-vision-camera.com/docs/guides/frame-processors-plugins-overview#long-running-frame-processors. It says that if your Frame Processor takes longer than a single frame interval to execute, or runs asynchronously, you can create a copy of the frame and dispatch the actual frame processing to a separate thread.

I understand that this is done to avoid blocking the main thread, which could cause the video feed to stutter. However, my WebRTC application does not need to process the frames in real time. I can simply send the frames to the server without any processing.

Therefore, I can use the vision camera to capture frames and then send them to the server using WebRTC. I will not be using the vision camera and WebRTC together, but rather sequentially.

In other words, you are using the vision camera to capture frames and then sending those frames to the server using WebRTC. You are not using the vision camera to process the frames in real time. This is perfectly acceptable, and there is no need to create a copy of the frame or dispatch the frame processing to a separate thread.

[8BallBomBom]

With alterations you could indeed have it where this module feeds frames over to the other module or vice versa but that would require a lot of compatibility work on both ends 🤔

Very much considered out of scope for this project.

[saghul]

I found this information on the react-native-vision-camera docs: https://react-native-vision-camera.com/docs/guides/frame-processors-plugins-overview#long-running-frame-processors. It says that if your Frame Processor takes longer than a single frame interval to execute, or runs asynchronously, you can create a copy of the frame and dispatch the actual frame processing to a separate thread.

I understand that this is done to avoid blocking the main thread, which could cause the video feed to stutter. However, my WebRTC application does not need to process the frames in real time. I can simply send the frames to the server without any processing.

Therefore, I can use the vision camera to capture frames and then send them to the server using WebRTC. I will not be using the vision camera and WebRTC together, but rather sequentially.

In other words, you are using the vision camera to capture frames and then sending those frames to the server using WebRTC. You are not using the vision camera to process the frames in real time. This is perfectly acceptable, and there is no need to create a copy of the frame or dispatch the frame processing to a separate thread.

There is no way for you to inject those frames into the WebRTC pipeline. You could send them over a datachannel I guess, but that will likely generate bad results since it's notvery performant, binary data needs to be base64 encoded.

[ciaoamigoschat]

Thank you for the responses. Let me explain what I need to do; perhaps there is another solution. I need to capture the image from the local camera every 60 seconds while in a video call. How can I achieve this using only WebRTC

[8BallBomBom]

You could use frame processors but it will require some native programming.
We do have Android support and there is a pr which needs pushing through to enable support for iOS over here.

[mskalra]

Hey @ciaoamigoschat I could be wrong, but I dont think you can achieve this with the existing WebRTC RN module on the device without making changes to native code.

You can do it pretty easily on the server side using that module, process the image there and then send it back to the client (assuming what you're building is not client to client).

Alternatively if you want to use react-native-vision-camera you can use frame processors to get the frame like @8BallBomBom suggested. But again you'll have to write native code to get the actual frame contents (https://github.com/yasintorun/vision-camera-base64 needs updating and is incompatible with v3 of vision-camera) and you'll also have to write your own native WebRTC component to send the video stream to your client/server.

TLDR: consider https://github.com/react-native-webrtc/react-native-webrtc and to process frames every 60 seconds on the receiving side.

[ciaoamigoschat]

Thanks for your consideration, but I'm interested in capturing the image on the transmitting side.

[saghul]

It is not possible with this plugin.