[iOS] getDisplayMedia() not working

[matallui]

I have this RN app which was setup via Expo SDK 46 and, after realizing screen capture is provided by this package, I've decided to give it a try.
I already had a working webrtc display app (using PeerJS), so I've created a simple RN client app with a button for obtaining a stream and connecting to the display and share it.

If I use getUserMedia({ video: true }) then I can see my camera feed being shared on the screen. However, if I use getDisplayMedia(), I get a blank stream.

The call to getDisplayMedia() doesn't seem to fail and I get a stream though:

{"_reactTag": "A15DC054-32BA-4A9C-96BD-F02CF5D79923", "_tracks": [{"_constraints": [Object], "_enabled": true, "_muted": false, "_settings": [Object], "id": "3384
03C0-2147-409F-90D8-4347ED9CC0E0", "kind": "video", "label": "338403C0-2147-409F-90D8-4347ED9CC0E0", "readyState": "live", "remote": false}], "active": true, "id": "A15
DC054-32BA-4A9C-96BD-F02CF5D79923"}

I also couldn't spot anything obvious on iOS syslog.

Expected Behavior

I would expect some sort of pop-up to ask for permissions to record screen, and then would expect to see my phone screen being shared on my display app.

Observed Behavior

Instead I get a blank stream, which reports 0 FPS on the display side.

Steps to reproduce the issue

• Create Expo app
• Install react-native-webrtc and @config-plugins/react-native-webrtc
• Configure config plugin in app.json
• Build development app via EAS and install on iPhone
• Run sample code (via expo start --dev-client)

Platform Information

• React Native Version: 0.69.4
• Plugin Version: 1.100.1
• OS: iOS
• OS Version: 15.6

[8BallBomBom]

Might want to have a look over here as the issue has been discussed before.
But what i can say is we could definitely add some information to the project docs similar to what we have for android.

[matallui]

@8BallBomBom thanks! I've seen that before but I still don't understand what to do to get it working.

If there are extra steps that need to be taken, there should be documentation backing this up, otherwise the feature is just unusable.

[saghul]

We need to document this better indeed. In the meantime: https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-ios-sdk/#screen-sharing-integration

[matallui]

@saghul Thanks!! Let me read through all of that and see if I can get something going...
Just a quick question: Will getDisplayMedia() work out of the box in Android? Or do we need some extra setup there as well?
I just managed to get my hands on an Android phone (running Android 11) and tried the same code. I do see the pop-up asking for permissions to record the screen, but I end up with a blank stream again.

[saghul]

On Android it will work out of the box but for Android >= 9 you need a foreground service with the MEDIA_PROJECTION permission.

[8BallBomBom]

There will be a better documentation up at some point but you can follow this if you plan to support Android 10+ screen sharing.

[matallui]

Ok, I managed to get Android screen sharing working after following the docs suggested by @8BallBomBom.
However, for iOS, even after following the Jitsi docs and adding/setting up the Broadcast extension, I still get the same result (no pop-up, blank stream).

There was just one step that I didn't do, related to JitsiMeetConferenceOption where they enable builder.setFeatureFlag("ios.screensharing.enabled", withBoolean: true). Not sure if this is something we need to do in a different way or what I could be missing.

As for the JS code, the only thing I'm doing is mediaDevices.getDisplayMedia(). Not sure if any other changes need to be made there.

Any ideas what I could be missing?

[matallui]

Finally got it to work on iOS. I guess I was missing the whole ScreenCapturePickerView logic.

Even though I got it working, a few notes:

• it's still not clear to me what the workflow for ScreenCapturePickerView is (in terms of what it exactly does and timings)
• the FPS on the stream are pretty low and, after some time, the stream stopped working (not the screen recorder) for no apparent reason (not sure if I did something wrong, or if it's just an issue)

Either way, thanks for the help!! I'll close this issue for now, but would like to see this better documented so everyone can understand exactly what to do to get screen capture working.

I'd be happy to help in whatever I can...

[saghul]

A PR with the steps you followed, adapted to a generic project would be welcome.

Note the low fps is intentional, since gDM is designed for mostly static content.

[matallui]

@saghul I'll get something going once I iron this out.

I've been looking into jitsi-meet code for reference and I noticed they don't really make any calls to mediaDevices.getDisplayMedia(). Does that mean that's not needed on iOS? Do you know how that works for them?

[saghul]

We do call it, but it's a few layers down, inside lib-jitsi-meet.

[matallui]

FYI, I've created this demo app which contains a couple of Expo config plugins to configure both iOS and Android projects for screen capture automatically. In the future I still intend to clean this up and make it more generic, but for now this is a good starting point.

[saghul]

I didn't know that was possible with Expo plugins, nice!

[Timson020]

@saghul @matallui

bro,
iOS screen sharing not work.
i see the doc
Jitsi screen sharing step

in the end, i dont know how to set JitsiMeetConferenceOptionsin the swift code

Can you help me. ?
Thank you very much !!!