Screen sharing showing black screen for local video on iOS

[agam-colaburate]

Expected behavior

The screen share i.e getDisplayMedia() should work as working on android i.e show the local screen

Observerd behavior

The screen sharing is not working on ios no matter real device or simulator, but works fine for the same code on android

Steps to reproduce the problem

create a dummy react native project, install the latest react native webrtc library and replace the app.js code with the code below

import React, {useState} from 'react';
import {
  SafeAreaView,
  StatusBar,
  Text,
  NativeModules,
} from 'react-native';
import {Colors} from 'react-native/Libraries/NewAppScreen';
import {
  RTCView,
  mediaDevices,
} from 'react-native-webrtc';

 const { ScreenShareFile } = NativeModules;

const App = () => {
  const [stream, setstream] = useState(undefined);
  
  const startShare = async () => {
    mediaDevices
      .getDisplayMedia({video: true})
      .then(handleSuccess, handleError);
  };

  const stopShare = async () => {
    mediaDevices.stopShare();
  };

  const handleError = (error) => {
    console.log("Error",error)
  };

  const handleSuccess = async stream => {
    setstream(stream);
  };

  const backgroundStyle = {
    backgroundColor: Colors.lighter,
    flex:1
  };

  return (
    <SafeAreaView style={backgroundStyle}>
      <StatusBar barStyle={'dark-content'} />

      <Text
        onPress={() => {
          startShare();
        }}
        style={{textAlign: 'center', width: '100%', marginVertical: 20}}>
        Press me to start sharing
      </Text>
      {stream && (
        <RTCView
          style={{
            borderColor: '#000',
            borderWidth: 1,
            width: '80%',
            height: '80%',
          }}
          streamURL={stream?.toURL()}
        />
      )}
      <Text
        onPress={() => {
          stopShare();
        }}
        style={{textAlign: 'center', width: '100%', marginVertical: 20}}>
        Press me to stop sharing
      </Text>
    </SafeAreaView>
  );
};

export default App;

Now run the app on iOS and click start recording, no pop up to start broad cast is shown and also, the screen or RTCView is black in color.

Please let me know if I have to do some extra setting to have this working in iOS. As of now, I just want to show the iOS local video only !

Platform information

iOS Real device

• React Native version: 0.65.1
• Plugin version: 1.92.1
• OS: iOS
• OS version: 14.3

[saghul]

Did you create your own broadcast extension? Unlike Android, screen-sharing does not work out of the box without cooperation from the app.

[agam-colaburate]

Yes I created my own broadcast extension with the following steps :
new > target > broadcast upload extension > filling out details , unchecking the ui required box .
Din't do anything apart from that.

I have also checked other threads in which they are referring to other pages for setting it up but its still not clear for a non iOS guy. i.e

step 1: add broadcast extension
step 2: config extension and copy files *.swift in sample project (as link https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-ios-sdk in title "TL;DR") and done for build extension
step 3: in react-native I use mediaDevices, MediaStream from 'react-native-webrtc for get displaymedia and mediasoupClient from 'mediasoup-client' for create send transport data (video streaming)
step 4: I add call show extension pop-up and mediaDevices.getDisplayMedia
So, I think in this way, I cannot getDisplayMedia from native source because I don't have media object bridge for displayMedia. do you have any suggestion

Can you please help ?
Though thanks for a prompt reply !

[saghul]

You need to follow the steps in here: https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-ios-sdk#screen-sharing-integration

Yes, it's native iOS, there is no way around that.

[agam-colaburate]

One more question,
The amount of code that I have written will work on a simulator if I follow all the steps carefully for iOS ?

[saghul]

This does not work on the simulator, you need a physical device.

[SabihaCetin]

@saghul

I have any question this issue.I followed this steps.

I cannot see share screen on my physical device. Actually , starting broadcast extensions, but I cannot see shared screen on my device's screen or participant's screen.

Do I have to run it in release mode? Or why cannot see ?

[saghul]

You should be able to run in debug mode, not using the Chrome debugger though.

[SabihaCetin]

@saghul thank you .
@agam-colaburate Did you find any solve about showing black screen bug ?

[singhagam1]

@SabihaCetin No luck so far, Tried a lot, the documentation on jitsi documentation are not that much clear for a non-iOS guy but still I tried them.
But unfortunately no success. I think @saghul if you have time, you should come up with a video/documentation for the repo and not redirect to jitsi ~ Just a suggestion

[irohitb]

@saghul Followed exactly the same steps, but getting red screen when I start broadcasting, any help on how to debug it?

[saghul]

Is any simulator involved?

[irohitb]

@saghul Nope.

Running it on a physical device in debug mode.

Also, I was looking at Jitsi code and I couldn't find reference to mediaDevices.getDisplayMedia in code?

I know, you can't debug my code but do you have any idea why someone could be seeing red-screen on physical device? I am using iPhone 12 mini running iOS 15. Jitsi App works fine.

[saghul]

No idea TBH. We run a vanilla version of this plugin, with no changes.

[irohitb]

@saghul got it.
One last thing (promise won't spam your notifications after this), as soon as I start screenshare, I get this in my xcode logs

server stream open completed

like these logs

2021-12-23 02:56:59.980625+0530 ClientMobile[12778:1168260] server stream open completed
2021-12-23 02:56:59.980701+0530 ClientMobile[12778:1168260] server stream open completed
2021-12-23 02:57:05.193274+0530 ClientMobile[12778:1167792] [connection] nw_socket_handle_socket_event [C13:1] Socket SO_ERROR [61: Connection refused]
2021-12-23 02:57:05.193503+0530 ClientMobile[12778:1167926] [connection] nw_connection_get_connected_socket [C13] Client called nw_connection_get_connected_socket on unconnected nw_connection
2021-12-23 02:57:05.193541+0530 ClientMobile[12778:1167926] TCP Conn 0x2839bb0d0 Failed : error 0:61 [61]

Does this point to anything?

[saghul]

Yep, that sounds like the broadcast extension is connecting to the wrong place. Did you correctly set the app group ID in the correct Info.plist file?

[singhagam1]

@irohitb
It would be really helpful for me as well as other devs if you can describe the implementation specially for ios because I am still struggling with the black screen.
TIA

[saghul]

It's described with great detail in the link here: #1071 (comment)

[irohitb]

@saghul Double checked everything on the app (app group seems to be correct).

From my broadcast extension logs, It seems to be failing here: https://github.com/jitsi/jitsi-meet-sdk-samples/blob/master/ios/swift-screensharing/JitsiSDKScreenSharingTest/Broadcast%20Extension/SocketConnection.swift#L143

Getting failure: -1

[saghul]

That means the listener was not ready yet. In what order are you running the operations?

[irohitb]

@saghul Yeah, My bad I was doing brute force approach and trying different things.

Here the new logs I am getting

open socket connection
client stream open completed
client stream open completed
client stream open completed
2021-12-25 18:31:42.506072+0530 broadcast[27414:2113234] fopen failed for data file: errno = 2 (No such file or directory)
2021-12-25 18:31:42.506226+0530 broadcast[27414:2113234] Errors found! Invalidating cache...
2021-12-25 18:31:42.583605+0530 broadcast[27414:2113235] fopen failed for data file: errno = 2 (No such file or directory)
2021-12-25 18:31:42.583728+0530 broadcast[27414:2113235] Errors found! Invalidating cache...

Made a small repo (with very minimal code) for the same as well
https://github.com/irohitb/screenshare-webrtc

[irohitb]

@saghul Couple of questions, what is this value?
https://github.com/jitsi/jitsi-meet-sdk-samples/blob/master/ios/swift-screensharing/JitsiSDKScreenSharingTest/Broadcast%20Extension/SampleUploader.swift#L30

And what it does? like what should someone set this to?

[saghul]

That's just a name assigned to the dispatch queue thread. You can set it to anything, it has no implications.

[irohitb]

@saghul
Thanks and one last question, In swift Screenshare sample app, value in info.plist for extension bundle identifier is set to

	<key>RTCScreenSharingExtension</key>
	<string>com.jitsi.example-screensharing.broadcast.extension</string>

https://github.com/jitsi/jitsi-meet-sdk-samples/blob/master/ios/swift-screensharing/JitsiSDKScreenSharingTest/JitsiSDKScreenSharingTest/Info.plist#L70

but in xcode project it shows this

Is this correct?

[peeratattt]

https://github.com/jitsi/jitsi-meet-sdk-samples/blob/master/ios/swift-screensharing/JitsiSDKScreenSharingTest/Broadcast%20Extension/SampleUploader.swift#L30

same, how to fix?

[agam-colaburate]

Hi @saghul

Following are the topics in the link, Can you let me know upto which topic are we supposed to follow ?
https://jitsi.github.io/handbook/docs/dev-guide/dev-guide-ios-sdk#screen-sharing-integration

1. Screen Sharing integration
2. Creating the Broadcast Upload Extension
3. Setting up the socket connection
4. Opening the socket connection
5. Sending video frames
6. Handling stop screen sharing

Should we follow all the above steps even if we are not using jitsi and only the webrtc plugin ?
I have been trying from quite long but got no success in IOS.
do we have a working example of screen-share with simple webrtc ? (specially for iOS)
And, for a frequently asked question - Yes, I am trying in a real device as it does not work on simulators.

and as simple as that, I am only trying to show the screen recorded video locally in the same screen.
Attaching for how it's working on android.

Another question,
is it important to do the below

`To enable data sharing, use Xcode or the Developer portal to enable app groups for the containing app and its contained app extensions. Next, register the app group in the portal and specify the app group to use in the containing app. To learn about working

with app groups, see Adding an App to an App Group.`

TIA

[saghul]

YEs, those steps apply to any app, not just jitsi.

[agam-colaburate]

We have to follow all the 6 steps ?

and again,
is it important to have the app created on the Apple developer just for development on your device also ?

[saghul]

Yes and yes.

[singhagam1]

Hey @saghul

I integrated the library and followed the steps mentioned in the jitsi meet link.
I am able to get the broadcast extension in the broadcast list of my PHYSICAL device also.
The problem that I am facing right now is that when I call the navigator.getDisplayMedia function, I am able to get the audio and transfer over the socket.
But, I have to manaully click on the record button from the native ios screen and start sharing to share the contents of the screen. In short, the broadcast pop up (screenshot) does not come show up by itself when I call the getDisplayMedia function.

Any clue of whats going wrong here ?
TIA.

[saghul]

What iOS version are you tearing on? Our trick to programmatically click the button seems to only work on iOS >= 14

[singhagam1]

@saghul

The iOS version that I am using is 15.1. For more refrence, if I add print("example") commands at any place in the copied files like samplehandler etc, they are never printed. Will they print ? Can this be the reason ? Can you tell me of any place where the print statement will surely print when I actually cal navigator. getDisplayMedia

"Our trick to programmatically click the button" -> Is this something that comes itself by the code of jitsi or we have to do this separately ?

[saghul]

You can do this: https://github.com/jitsi/jitsi-meet/blob/d651ecb166fdd5b3914614b2be2a183876496e64/react/features/toolbox/components/native/ScreenSharingIosButton.js#L86

Create your own button, and when the user taps it do the same thing we do to show the system picker.

[singhagam1]

Ok, Thanks,
I will try this.

[singhagam1]

That really worked @saghul
Thanks alot,
One last question, can we convert it into a promise so that I can assure that if the user has started broadcast, then only I should connect him to the socket.

[saghul]

There is a retry mechanism, it shouldn't matter. Are you running into issues?

[singhagam1]

@saghul Not running into an issue, basically,

I call "NativeModules.ScreenCapturePickerViewManager.show(handle);" and Navigator.getdiplaymedia at the same time and in the success of Navigator.getdiplaymedia, I connect to the socket.

Now there is a scenario, lets say I click the start streaming button and both the native picker and the navigator.displaymedia are called and I might land into a situation where the user does not click on the start broadcast button from the picker but the user actually is connected to the socket with a black screen shown to the other users, untill he clicks the black screen.
basically How can I check if the user has actually clicked the start broadcast button or not so that I can proceed or take any relevant decision.
Hope I make sense.

[saghul]

I think I understand. I don't think we currently have a way to know when the user pressed the button, alas.

I mean, we do know it on the native side, but not in JS. A patch would be welcome.

[singhagam1]

@saghul
I would love to try it though I dont have much iOS knowledge because this is a really important thing to have.

1. Can you tell me which file in the native file will have this function to know if the broadcast has started/user has clicked the start broadcast ?
2. Do we have a method to stop the broadcast programatically ? Because right now when I kill the stream, I have to manually stop the broadcast. (like click on the red notification bar and stop.)

[saghul]

Ah wait. While looking into answer you I realized how we do it :-)

In the app part, we post notifications when the user pressed the start button, when as a result of that we create the track: https://github.com/jitsi/jitsi-meet/blob/8456a63a23fdfc1b00a5115e9bb609b1722def5f/ios/sdk/src/ScheenshareEventEmiter.m#L60

[singhagam1]

Wow thats great, I will give it a try, Thanks man.

[singhagam1]

@saghul
I gave it a try but I think there is no such function in react native webrtc to achieve this, is my assumption correct ? and this is still to be developed for react native webrtc ?

[saghul]

You need some native code yes.

[singhagam1]

Hey @saghul
One quick question, Do you have a sample which would help me integrate this functionality in webrtc ?
TIA.

[saghul]

What I linked earlier. It's what we use in Jitsi Meet.

[singhagam1]

Hey @saghul Sorry for bothering again, but I am trying to achieve the maximum I can from the library and will try to help people in future also who face issue in this library.
I have a couple of questions :

lets say I have a button on the react native side to stop the screen share/streaming etc , how do I stop the iOS native side replaykit recording ( the red notification bar - which we have to click manually and click stop broadcast )
I used a hack to get the event of when a person hits the start broadcast on the native broadcast pop up but How do I hide it from the react native part ? do we have something equivalent to .show() that can hide it ? like .hide().
TIA.

[saghul]

Unfortunately I don't think that's possible.

[TarasDumych]

Hey @irohitb. Did you resolve red screen issue? will be appreciative for answer

[irohitb]

Hey @TarasDumych

Unfortunately no,

Haven't looked at in a while but will try it over the weekend again.

[peeratattt]

@singhagam1 How to implement to know if the broadcast has started/user has clicked the start broadcast ?
Please, share for me

[fukemy]

@peeratattt did u find solution?

[YINGCHAO-98]

@irohitb

@saghul是的，不好的是我正在使用实力方法并尝试不同的东西方。

这是我得到的日记

open socket connection
client stream open completed
client stream open completed
client stream open completed
2021-12-25 18:31:42.506072+0530 broadcast[27414:2113234] fopen failed for data file: errno = 2 (No such file or directory)
2021-12-25 18:31:42.506226+0530 broadcast[27414:2113234] Errors found! Invalidating cache...
2021-12-25 18:31:42.583605+0530 broadcast[27414:2113235] fopen failed for data file: errno = 2 (No such file or directory)
2021-12-25 18:31:42.583728+0530 broadcast[27414:2113235] Errors found! Invalidating cache...

为同样的事情做了一个小回购（代码非常少） https://github.com/irohitb/screenshare-webrtc

Hi brother, have you found a solution for this? Can you teach me