unable to get remote stream in case when a “remote stream” is attached by calling “peer.addStream(remoteStream) #2

Open
muaz-khan opened this Issue Feb 21, 2013 · 37 comments

Projects

None yet
@muaz-khan
Owner

Issue: unable to get remote stream in case when a “remote stream” is attached by calling “peer.addStream ( remoteStream )”

Is it beneficial to attach remote stream?

Yeah, it allows us overcome burden from a single peer. Burden will be shared.

Also, we can support a wide range of peer’s connectivity.

How to attach remote stream?

var MediaStream = window.webkitMediaStream || window.MediaStream;

firstPeer.onaddstream = function(remoteSteam) {
      remoteStream = new MediaStream(remoteSteam.audioTracks, remoteSteam.videoTracks);
      otherPeer.addStream(remoteStream);  /* attaching remote stream */
};

What I want to do?

WebRTC -Experiment

  1. First peer will handle first three peers
  2. 2nd peer will handle next three peers (5, 6, 7)
  3. 5th peer will handle next three peers (8,9,10)
  4. 8th peer will handle next three peers (11, 12, 13)
  5. And so on.

We will get following benefits (in case of success):

  1. Video will never freeze for 11th and upper peers
  2. 1st peer don’t need to handle all participants
  3. You don’t need to install/buy a middle server (like Asterisk) for small projects (to support a few hundred peers’ connectivity)

A known bug in chromium: peer.onicecandidate not fires for 11th peer. You don’t need to worry about that bug!!

Pitfall: If peer number 1, 2, 5, 8 … tries to leave the room without informing other peers to play a host role.

WebRTC -Experiment

If peer number 2 leaves the room, without informing master peer or peer number 5 to handling upcoming peers.

A demo experiment to test it:

https://googledrive.com/host/0B6GWd_dUUTT8V1Fodm9WQldkb28/

Open 4 tabs....1st table should create room……..join room from other tabs.....see the behavior of the 4th tab.

@muaz-khan
Owner

unable to generate ice candidates (on the answerer side) when attaching remote stream..

a=candidate:2437072876 1 udp 2113937151 192.168.1.2 54082 typ host generation 0
a=candidate:2437072876 2 udp 2113937151 192.168.1.2 54082 typ host generation 0
a=candidate:2437072876 1 udp 2113937151 192.168.1.2 54082 typ host generation 0
a=candidate:2437072876 2 udp 2113937151 192.168.1.2 54082 typ host generation 0
a=candidate:941443129 1 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0
a=candidate:941443129 2 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0
a=candidate:941443129 1 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0
a=candidate:941443129 2 udp 1845501695 39.47.69.177 10136 typ srflx raddr 192.168.1.2 rport 54082 generation 0

--------offer sdp provided by offerer

v=0
o=- 2216700829 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video
a=msid-semantic: WMS
m=audio 1 RTP/SAVPF 103 104 111 0 8 107 106 105 13 126
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:Yy0jsw+XcU7Ovmw4
a=ice-pwd:9kjBUxNEoOOO0yViUNEH9Yhe
a=ice-options:google-ice
a=fingerprint:sha-256 ED:B9:CF:19:5F:A3:F4:D3:31:FC:F5:18:3C:AF:60:9E:B1:52:A5:32:69:8A:22:63:05:AD:98:93:ED:33:2A:7E
a=recvonly
a=mid:audio
a=rtcp-mux
a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:CFkD+WaozWTFg41l4mi5rdoAscTYMJa8CB1HgVpW
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:107 CN/48000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60
m=video 1 RTP/SAVPF 100 116 117
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:Yy0jsw+XcU7Ovmw4
a=ice-pwd:9kjBUxNEoOOO0yViUNEH9Yhe
a=ice-options:google-ice
a=fingerprint:sha-256 ED:B9:CF:19:5F:A3:F4:D3:31:FC:F5:18:3C:AF:60:9E:B1:52:A5:32:69:8A:22:63:05:AD:98:93:ED:33:2A:7E
a=recvonly
a=mid:video
a=rtcp-mux
a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:CFkD+WaozWTFg41l4mi5rdoAscTYMJa8CB1HgVpW
a=rtpmap:100 VP8/90000
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
RTCPeerConnection-v1.4.js:191

--------answer

sdp: v=0
o=- 315182876 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video
a=msid-semantic: WMS
m=audio 1 RTP/SAVPF 103 104 111 0 8 107 106 105 13 126
a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:Do0n1o1bjqhwgfo7jsfqBi2pgBx9i9egkjngrjkf
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:ZQglAeIL9/0MtR+R
a=ice-pwd:gbAu8knZaPhZrBQc9mDjK2AT
a=fingerprint:sha-256 50:DD:32:5B:31:92:C0:CF:77:2A:DC:66:45:29:6C:58:73:A8:77:AF:10:38:D0:FA:53:D9:55:26:0A:DC:B2:6A
a=recvonly
a=mid:audio
a=rtcp-mux
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:107 CN/48000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60
m=video 1 RTP/SAVPF 100 116 117
a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:Do0n1o1bjqhwgfo7jsfqBi2pgBx9i9egkjngrjkf
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:ZQglAeIL9/0MtR+R
a=ice-pwd:gbAu8knZaPhZrBQc9mDjK2AT
a=fingerprint:sha-256 50:DD:32:5B:31:92:C0:CF:77:2A:DC:66:45:29:6C:58:73:A8:77:AF:10:38:D0:FA:53:D9:55:26:0A:DC:B2:6A
a=recvonly
a=mid:video
a=rtcp-mux
a=rtpmap:100 VP8/90000
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000

Maybe RTP ports already opened....We may need to change them dynamically.

@muaz-khan
Owner

Latest one-page demo on attaching remote media streams. Still fails.

@peili
peili commented Apr 15, 2013

Any process on this issue so far?

@muaz-khan
Owner

This issue is not fixed yet. Attachment of remote media streams still failing on both chrome and firefox. I tried audio-only remote streams too but no success.

@peili
peili commented Jun 24, 2013

Do you know when Google/Mozilla are going to support this feature? I couldn't find clear informations on the roadmaps..

@muaz-khan
Owner

I'm not sure, however it seems that audio-only remote streams' attachment works fine.

@muaz-khan
Owner
A general comment is that we only support one audio source today in Chrome given that we only have one ADM. To me it sounds like we need a new type of source here where the audio is driven by the remote peer. Perhaps it would be less complicated to build something initially if we only worked in relay mode (one source).

Reference: https://code.google.com/p/webrtc/issues/detail?id=383

.....take a remote stream from one peer connection and adding it to another peer connection and having it Just Work™ for that second peer.

Implementation awaited!

@RonMen
RonMen commented Jul 26, 2013

Hi @muaz-khan,

I think this is what I thought about on my other issue questioning. The use case is:

1.) Peer1 connects with audio only
2.) Peer2 connects with audio only
... later on one/both switch
3.) Peer1/Peer2 switches over to remove audio only stream and instead add a video+audio stream

In this case I get a black video on the peer this stream is delivered to sending a "new" offer and there are removeStream + addStream events fired up but also the data of the newly added stream isn't send.

I tried multiple settings and also tried to find an answer but it doesn't seem to work. Also the sample implementation presented in the webrtc issue tracker isn't working like expected: https://code.google.com/p/webrtc/issues/detail?id=1086

Do you have any idea how to bring this up and running or is there something missing in browsers webrtc implementation for now?

@muaz-khan
Owner

Try this demo: Switching streams from screen-sharing to audio+video at runtime. (Renegotiation)

  1. Both peers are sharing screen
  2. You can switch to audio+video stream; old stream (i.e. screen-sharing) will be removed.

Don't need to add ice candidates when renegotiating.

@RonMen
RonMen commented Jul 28, 2013

Dear @muaz-khan, thx for your sample. I also found something else since setting the localDescription failed while switching like explained here: https://groups.google.com/forum/#!msg/discuss-webrtc/ets-eex6H-0/bkN2UwZI-gEJ

I changed the {optional: [{DtlsSrtpKeyAgreement: true}]} of the peerconnection and I tried without it and the streams are switched well now.

@skanagavelu

We need this idea. This communication is low cost for end user.
Mauz; How this is supported for screen share?

@skanagavelu

Hi Muaz,

In your example below

firstPeer.onaddstream = function(remoteSteam) {
remoteStream = new MediaStream(remoteSteam.audioTracks, remoteSteam.videoTracks);
otherPeer.addStream(remoteStream); /* attaching remote stream */
};

Why we are recreating remoteStream again using new MediaStream ?

And pls tell me what we have to do for screen share while creating new instance of MediaStream...?

@muaz-khan
Owner

There are some bugs preventing video tracks to be forwarded. In screen sharing apps; we need to retransmit video RTP packets; which seems failing on both chrome and firefox.

In the example; new media stream is created; because in old days, chrome were throwing not-supported exceptions for MediaStream because only LocalMediaStream objects were permitted to be attached using peer.addStream method.

Nowadays, you can attach remote stream directly, on chrome; without initiating new object; because both local and remote streams have identical MediaStream type.

I think identical ports (RTP or UDP) are causing failures to attach remote media streams. Though, I don't know the exact issue. Maybe ICE Agent is unable to locate right ports to retransmit remote media stream.

Maximum peer connections limit is 256; so it is preferred to use application specific bandwidth parameters to stream low-quality packets; also specify width/height constraints to handle freezing issues.

@peili
peili commented Sep 3, 2013

Any updates on "attaching remote audio/video stream"?
Audio seems to work. Video not?

@abhishanksahu

Hi Muaz, Is there any progress in this issue? I am able to attach remote streaming on the same page. But when I try to do the same over two different pages it is failing.

@muaz-khan
Owner

It works. Try following demo on chrome stable (desktop):

<script src="//www.webrtc-experiment.com/RTCMultiConnection-v1.6.js"> </script>

<button id="open-main-session">open main session</button>
<button id="forward-main-session">forward main session</button><br />

<button id="join-main-session">join main session</button>
<button id="join-forwarded-session">join forwarded session</button>
<script>
var h1 = document.createElement('h1');
h1.innerHTML = 'user-id: ' + prompt('What is Your Name');
document.body.insertBefore(h1, document.body.firstChild);

// http://www.rtcmulticonnection.org/docs/constructor/
var mainConnection = new RTCMultiConnection();

// http://www.rtcmulticonnection.org/docs/session/
mainConnection.session = {
    video: true,
    oneway: true
};

// http://www.rtcmulticonnection.org/docs/constructor/
var dummyConnection = new RTCMultiConnection('dummy-connection');

// http://www.rtcmulticonnection.org/docs/session/
dummyConnection.session = {
    video: true,
    oneway: true
};

// http://www.rtcmulticonnection.org/docs/dontAttachStream/
dummyConnection.dontAttachStream = true;

// http://www.rtcmulticonnection.org/docs/onstream/
mainConnection.onstream = function(e) {
    // see this line--------------------------
    dummyConnection.attachStreams.push(e.stream);
    document.body.appendChild(e.mediaElement);
};

// http://www.rtcmulticonnection.org/docs/onstream/
dummyConnection.onstream = function(e) {
    alert('Wow, got forwarded stream!');
    document.body.appendChild(e.mediaElement);
};

document.querySelector('#join-main-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/connect/
    mainConnection.connect();
};

document.querySelector('#join-forwarded-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/connect/
    dummyConnection.connect();
};

document.querySelector('#open-main-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/open/
    mainConnection.open();
};

document.querySelector('#forward-main-session').onclick = function() {
    this.disabled = true;

    // http://www.rtcmulticonnection.org/docs/open/
    dummyConnection.open();
};
</script>

dummyConnection is used to forward media stream. Main forwarding stuff happened here:

mainConnection.onstream = function(e) {
    dummyConnection.attachStreams.push(e.stream);
};
@abhishanksahu
abhishanksahu commented Feb 5, 2014 edited

Hi Muaz,

Thanks for your help, I tried this demo and its work fine for video, but
when I enable audio also, on the forwarded stream audio is missing, there
is no sound from the third tab. Can you tell me why?

Thank You

@muaz-khan
Owner

Did you appended audio:true in the session object?

mainConnection.session = {
    audio: true,
    video: true,
    oneway: true
};
@abhishanksahu

Yeah I did that, on the second tab I am able to listen audio, but when on
third tab I join the forwarded session, I can only see the video and there
is no sound coming from it. With latest version of chrome u can see which
tab is playing audio right? There was no such sign over this tab.

On 5 February 2014 22:52, Muaz Khan notifications@github.com wrote:

Did you appended audio:true in the sessionhttp://www.rtcmulticonnection.org/docs/session/object?

mainConnection.session = {
audio: true,
video: true,
oneway: true};

Reply to this email directly or view it on GitHubhttps://github.com/muaz-khan/WebRTC-Experiment/issues/2#issuecomment-34213645
.

@harshg0910

@muaz-khan
@abhishanksahu
I did some experiments with audio only :
Computer 1 :

  1. Opened the main session in one tab
  2. Joined the main session and forwarded it in the other tab of same computer.
    Computer 2
  3. Joined the forwarded session but the audio player was there with no sound.
  4. Joined the main session in other tab and I was able to hear the sound.
  5. Now the tab containing the forwarded stream was also playing the sound ( I checked it by pausing the sound of the main session tab).
  6. I close the tab with main session, the sound from the forwarded session also dies.

Seems interesting as I am unable to find the reason for failure of forwarded stream at the same time its relation with main stream.

@pandalaw
pandalaw commented Feb 7, 2014

Hi Muaz,

I tried to forward stream of screen, but it failed. Can you tell me why?
Computer 2 cannot get the screen of computer 1 as well.

Thank You

@muaz-khan
Owner

I tried this demo; changed this line:

mainConnection.session = {
    screen: true,
    oneway: true
};

It worked when I shared screen from canary. In the below gif; you can see that I tested between chrome canary, chrome stable and opera next 19.0.

@abhishanksahu

Hi Muaz,

Did you find any remedy for the problem I described you (no sound from the
forwarded stream).

Thanks.

On 5 February 2014 22:56, Abhishank Sahu abhishanksahu@gmail.com wrote:

Yeah I did that, on the second tab I am able to listen audio, but when on
third tab I join the forwarded session, I can only see the video and there
is no sound coming from it. With latest version of chrome u can see which
tab is playing audio right? There was no such sign over this tab.

On 5 February 2014 22:52, Muaz Khan notifications@github.com wrote:

Did you appended audio:true in the sessionhttp://www.rtcmulticonnection.org/docs/session/object?

mainConnection.session = {
audio: true,
video: true,
oneway: true};

Reply to this email directly or view it on GitHubhttps://github.com/muaz-khan/WebRTC-Experiment/issues/2#issuecomment-34213645
.

@muaz-khan
Owner

See this comment:

Due to some architecture problems in the chromium remote audio track implementation, today we are always mixing the data in WebRtc, and passing the mixed data to Chrome. This ways prevents us getting the correct stream for the specific audio track, and it also blocks us feeding the correct remote audio track data to the peer connection.

After talking to a couple of colleagues about this feature, we decided to postpone this feature until we fix the architecture problems.

That's why remote audio recording doesn't work.

@pandalaw
pandalaw commented Mar 3, 2014

Hi Muaz,

I tried this demo. and changed line 135

mainConnection.session = {
    video: true,
    audio: true,
    screen: true,
    oneway: true
};

When I joined forwarded session, I can get screen stream only.

Please help me to find out the problem :)

Thank you.

@muaz-khan
Owner

After looking into logs; it is appeared that only single "remote" media stream is attached to peer connection object:

rtcmulticonnection-remote-stream-forwarding-1

In theory, it MUST attach both audio+video and screen streams. You can see that we are trying to attach both:

rtcmulticonnection-remote-stream-forwarding-2

After going a little bit deeper; it is appeared that both remote streams has identical ids and labels. It can be RTCMultiConnection implementation issue.

@shtefcs
shtefcs commented Apr 1, 2015

Hi Muaz,

Is this problem fixed by chrome guys ?

@josephfrazier

@shtefcs, it's been quite a while since you asked, but yes, Chrome 49 (Canary, at the time of writing) should:

@spozitor

Any peer output without a "closed" sign, causes the remaining peers seek to reconnect?
You can 100,000 users?

@maurodelazeri

what happens with the pear 60 when pear 59 was disconnected ?

@muaz-khan
Owner

If peer-59 leaves, peer-60 is told to rejoin-broadcast. This is exactly how RTCMultiConnection is currently supporting it.

If peer-60 will rejoin, server will give him fresh relaying-peer-id. During this rejoin-process, RTCMultiConnection tries to play recorded stream until fresh stream-event is fired.

If peer-5 leaves then all-kinds-of nested child are told to rejoin. Currently this mechanism hangs the browser. Better solution will be only top nested child should be asked to rejoin; however other deeper nested child should renegotiate connections to get fresh stream(s).

@mistris mistris referenced this issue in muaz-khan/RTCMultiConnection Feb 8, 2016
Open

RTCMultiConnection v3 Video Scalable Broadcast #88

@gatherhub

Hi Muaz, I read through this thread and run your demo it seems that adding remote stream to another peerconnection does work. Howerver, when I wrote my own code, I cached the remote stream and added it to another peerconnection, the far end does not get onaddstream event fired and cannot get the forwarded stream. There was no errors and iceconnectionstate is completed too. I have also checked the ice candidates information. There are all generated completely and correctly without duplication to the original peerconnection. Would you help to provide some hint? Thank you very much!

@atsepkov
atsepkov commented Sep 1, 2016 edited

@muaz-khan For solving the problem you describe with peer-5 leaving, could the server instead tell peer 6 or 7 (who should be in the same link in the chain as 5) to take over instead? Similarly, could this system be made redundant/safe by having each child peer be aware of all 3 parent peers in the chain and fallback automatically if one drops?

@visionsTeam

@muaz-khan
Hello,
I have successfully implemented your code on my php website and its very useful really appreciative. But the only issue i am facing is that when one of the user is on chrome browser and other in mozilla
when they share their webcam the video freeze and one of the user automatically left the room.
it will be very kind if you suggest some.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment