First experiments with ICE restarts

[lminiero]

This PR tries to lay the foundation to add support for something that would be quite helpful: ICE restarts. These are typically needed whenever something in your network changes (e.g., you move from WiFi to mobile or a different WiFi) but want to keep the conversation going: in this case, an ICE restart needs to take place, as the peers need to exchange the new candidates they can be reached on. Of course, since Janus always stays where it is, this is only relevant for users. Beware that ICE restarts are not enough to cover the scenario I mentioned: in fact, if your signalling channel going down due to the network switch results in the PeerConnection being destroyed on Janus (what happens for instance with the WebSockets transport in Janus itself), then ICE restarts can't help. If you're wrapping the Janus API, you need to make sure that signalling can be kept alive and restored after a switch. How that happens and how to distinguish that from plain timeouts is up to you: this PR only covers media once you decide that happened.

Before I begin, let me anticipate that this is NOT ready and complete. While the core bits have been done, and ICE is something the core itself worries about, this is not a feature that is transparent to plugins, since they need to be aware a renegotiation is taking place (e.g., to increase the SDP version). This is especially true in case we also decide to allow offerer/answerer to switch roles during a renegotiation (more on this later).

I conceived this PR as a patch to #403 (reference counters) and not master as I want to accellerate the adoption of those as well. This may be a good opportunity for interested users to start messing with that too.

Intro

This patch only modifies EchoTest and Streaming plugins/demos to test the feature. The reason for this is simple: they're the simplest plugins to involve to test the different roles Janus can take. In the EchoTest, the user always offers and Janus always answers; in the Streaming plugin it's the other way around. For this reasons, they both allowed me to experiment a bit.

Besides, the patch modifies janus.js as well to allow for ICE restarts to occur. In particular, when doing a createOffer you can now add an iceRestart boolean property to force an ICE restart to be originated: this results in new ICE credentials, which are detected by Janus, which as a consequence results in an ICE result taking place. Should Janus issue an ICE restart, instead, it should be enough to just answer and the browser does the rest.

That said, as anticipated one of the main questions I asked myself was: should we allow users to always originate a new offer to restart ICE, whether the handle supports that or not (see the Streaming plugin or VideoRoom subscribers, for instance), or should we stick to the approach the plugin uses, and let restarts be triggered by users in another way (e.g., a new plugin message or changes to existing ones)? I tried to address this in the PR by allowing both in the Streaming plugin. I'm personally in favour of the latter (keeping roles) as it simplifies things both on the server and the client side, as it will be clearer in the next examples.

Echo Test

In order to force an ICE restart during an echotest demo, open the JavaScript console and paste this:

echotest.createOffer({
    media:{data:true},
    iceRestart:true,
    success:function(jsep) {
        echotest.send({"message": {bitrate: 128000}, "jsep": jsep});
    }
});

This is basically the same thing as when we create the new PeerConnection in the demo, with an additional iceRestart:true. If you monitor the Admin API demo page, this should result in new ICE credentials in the SDP, new candidates being exchanged, and a change in the selected pair.

Streaming (sending offer instead of answer)

In order to force an ICE restart during a streaming demo (I used the radio broadcast for my tests), open the JavaScript console and paste this:

streaming.createOffer({
    media:{audioRecv:true, audioSend:false, video: false},
    iceRestart:true,
    success:function(jsep) {
        streaming.send({"message": {request: "watch", id: 2}, "jsep": jsep});
    }
});

As you can see, here we can't just set iceRestart:true, but we also have to mess with the media, which means being aware of what was negotiated and the directions (which means manually inspecting the old SDP). Here we have to explicitly specify we want recvonly audio and no video, or otherwise the updated SDP will contain unsupported (and useless, considering the scenario) changes to the session.

To carry the new SDP, we simply send a {request: "watch", id: 2} again: the plugin has been modified to know it's not a new subscription, but an update to an existing one. Anyway, the createOffer thing feels awkward here.

Again, the Admin API or webrtc-internals will show the changes.

Streaming (Janus sends updated offer)

To keep the same behaviour the plugin uses for new streams, we can have Janus originate an ICE restart with an updated offer. Of course, it's still up to the user to state their intention to restart. In order to force the Janus-originated ICE restart during a streaming demo (again, I used the radio broadcast for my tests), open the JavaScript console and paste this:

streaming.send({
    "message": {
        request: "watch", id: 2, refresh: true
    }
});

So, exactly the same watch we sent initially, but with a refresh:true property the plugin has been condifured to understand. When refresh:true is there, the plugin updates the SDP version and sends an offer as usual, but clarifying it's an update. The core issues an ICE restart and sends the offer to the user. The same code already in place handles the offer and prepares an answer, that automatically restarts things.

Again, the Admin API or webrtc-internals will show the changes.

Conclusions

Please test and provide feedback, expecially on what we should support as a mechanism. Allowing users to offer may be helpful, but awkward on the user side, and not transparent at all to plugins, which need to be implemented to support both approaches even though they only use one (see examples above).

Should your tests on this preliminary effort work as expected, I'll proceed to extend support to other plugins as well for further experimentation.

[lminiero]

Since I got no feedback at all, I went on by myself. Just worked to add this new ICE support to other plugins: right now only the SIP plugin is missing (as we have to take into account possible updates on the SIP side as well so I want to think better about this). I also changed the way the Streaming plugin works, which now only uses the server-originated offer of the examples I made before.

Summarizing below the behaviour for each plugin and how you can test it (again, getting Admin API info on the updated handle should display a new selected-pair).

EchoTest

Nothing changed from the original commit. Just add iceRestart:true to a new createOffer and it will trigger an ICE restart. Handle answer as usual.

echotest.createOffer({
	iceRestart: true,
	media:{ data: true },
	success:function(jsep) {
		echotest.send({message: {bitrate: 128000}, jsep: jsep});
	}
});

Streaming

To trigger an ICE restart, send a new watch request for the same ID you were already watching, and add refresh:true. This will result in a new offer from Janus, that you can answer the usual way.

streaming.send({
	message: {
		request: "watch", id: 2, refresh: true
	}
});

VideoCall

Whether you're the caller or the callee, you can force an ICE restart by sending a new offer, and attach it to a set request. This SDP will not be forwarded by the peer, but will stay between you and Janus. Obviously the media will have to contain the same stuff you negotiated originally (in this case there's a data:true because we use datachannels in the demo). Handle answer as usual.

videocall.createOffer({
	iceRestart: true,
	media: { data: true },
	success: function(jsep) {
		var body = {request: "set"};
		videocall.send({message: body, jsep: jsep});
	}
});

AudioBridge

To trigger an ICE restart, create a new offer with iceRestart:true and attach it to a configure that includes a refresh:true (actually not needed, but cleaner from a semantics perspective). Handle answer as usual.

mixertest.createOffer({
	iceRestart: true,
	media: { video: false },
	success: function(jsep) {
		var publish = {request: "configure", refresh: true};
		mixertest.send({message: publish, jsep: jsep});
	}
});

VideoRoom

Different behaviour in case you're updating a publisher or a subscriber.

For publishers, create a new offer with iceRestart:true and attach it to a configure that includes a refresh:true (actually not needed, but cleaner from a semantics perspective). Handle answer as usual.

sfutest.createOffer({
	iceRestart: true,
	media: { audioRecv: false, videoRecv: false, audioSend: true, videoSend: true },
	success: function(jsep) {
		var publish = {request: "configure", refresh: true };
		sfutest.send({message: publish, jsep: jsep});
	}
});

For subscribers, just send a configure with a refresh:true property on the subscriber handle to trigger an ICE restart. As in the Streaming plugin, it will result in a new offer from Janus, that you can answer the usual way.

feeds[1].send({message: {request: "configure", refresh: true}});

TextRoom

To force an ICE restart, send a refresh request (new command that didn't exist before) to the plugin. As in the Streaming plugin, it will result in a new offer from Janus, that you can answer the usual way.

textroom.send({message:{request:"refresh"}});

Record&Play

Different behaviour in case you're recording or replaying something.

For recorders, create a new offer with iceRestart:true and attach it to a record request, pretty much as you'd do to start a new one. Handle answer as usual.

recordplay.createOffer({
	iceRestart: true,
	success: function(jsep) {
		var body = {request: "record", name: myname };
		recordplay.send({message: body, jsep: jsep});
	}
});

For playout, send a play as you did originally, but with a refresh:true property, on the playout handle to trigger an ICE restart. As in the Streaming plugin, it will result in a new offer from Janus, that you can answer the usual way.

var play = { request: "play", id: parseInt(selectedRecording), refresh: true };
recordplay.send({"message": play});

VoiceMail

Pretty meaningless considering all VoiceMail sessions last at max 10 seconds, but anyway... To force an ICE restart, create a new offer which is the same as the original one but with iceRestart:true, and attach it to a refresh request (new command in this plugin). Handle answer as usual.

vmailtest.createOffer({
	iceRestart: true,
	media: { audioRecv: false, video: false},
	success: function(jsep) {
		var publish = { request: "refresh" };
		vmailtest.send({"message": publish, "jsep": jsep});
	}
});

SIP plugin

TBD.

Conclusions

Hope that now that almost all plugins are updated, I'll get some feedback on this. Please test!

[AjayChoudary]

Great work @lminiero 👍, its a most awaited feature in Janus to handle internet fluctuations.

But i tested it in a different scenario and facing few issues !

Scenario: Switching steam in active peerconnection (Common use case for switching microphones/webcams or webcam to screenshare).

Changes in the client side :

Created new stream and added it to peerconnection after removing old stream.
Created the offer (without ice_restart constraint) and sent to janus
Set the answer to pc received from janus.
Then client started sending new stream to janus.

Observations in Janus Server:
1> Janus is dropping the RTP packets with new SSRC as it is not updating the audio/video_ssrc_peer .
Fixed it by resetting the SSRC value here before processing new offer .

2> While processing new RTP getting below warning logs but sequence number is resetting (hopefully no issue with this)
[Sat Feb 11 16:15:44 2017] [WARN] [8598208881573407] Negotiation update, checking what changed...
[Sat Feb 11 16:15:44 2017] [WARN] [8598208881573407] Big sequence number jump 11293 -> 6821 (audio stream)
/ Big sequence number jump 21369 -> 11928 (video stream)

3> Need to test recording stream with multiple ssrc's.

4> While sending to listeners mostly getting below srtp protect error and audio is always audible at listeners. (something we need to handle similar to switch stream)

[Sat Feb 11 16:15:44 2017] [ERR] [ice.c:janus_ice_send_thread:3567] [1012232238932247] ... SRTP protect error... err_status_replay_old (len=100-->100, ts=2048199192, seq=4110)...
[Sat Feb 11 16:15:44 2017] [ERR] [ice.c:janus_ice_send_thread:3567] [1012232238932247] ... SRTP protect error... err_status_replay_old (len=99-->99, ts=2048200152, seq=4111)...
[Sat Feb 11 16:15:44 2017] [ERR] [ice.c:janus_ice_send_thread:3567] [1012232238932247] ... SRTP protect error... err_status_replay_old (len=104-->104, ts=2048201112, seq=4112)...
[Sat Feb 11 16:15:44 2017] [ERR] [ice.c:janus_ice_send_thread:3567] [1012232238932247] ... SRTP protect error... err_status_replay_old (len=93-->93, ts=2048202072, seq=4113)...

[lminiero]

This PR doesn't handle renegotiations just yet, only ICE restarts. One step at a time...

[chest3x]

Hello, I've been testing this branch on recordplay plugin.
I am experiencing a segmentation fault when I'm trying to Record myself for the 2nd time on the same session.

Log from GDB:
http://pastebin.com/55XcRNQh

[lminiero]

@chest3x not sure about what you were trying to do: I see you closed a recording there, but then tried to do an ICE restart for the second session? That's not what ICE restarts do: they allow you to restart the ICE state machine while you're on a recording, in this case, not to start a new recording. For that you need to do a second, new, negotiation (so brand new offer/answer).

Can you elaborate on what you tried to do? In any case, you probably found a bug we need to fix.

[lminiero]

I realize now that the examples I wrote may have been misleading. When I wrote "create a new offer with iceRestart:true and attach it to a record request, pretty much as you'd do to start a new one":

recordplay.createOffer({
	iceRestart: true,
	success: function(jsep) {
		var body = {request: "record", name: myname };
		recordplay.send({message: body, jsep: jsep});
	}
});

I meant as an update to the recording you started: the name must be the same, you can't restart and change recording name. Maybe it would be a good idea to add a refresh:true to that as well to make it clear we're refreshing an existing recording session?

[lminiero]

This commit should fix your crash, let me know if that's not the case.

[lminiero]

This last commit adds ICE restart support to the SIP plugin too. To force an ICE restart, generate a new offer that forces an ICE restart and attach it to a new request called "update":

sipcall.createOffer({
	iceRestart: true,
	media: { video: false },
	success: function(jsep) {
		var update = {request: "update"};
		sipcall.send({message: update, jsep: jsep});
	}
});

On the plugin side it generates a SIP re-INVITE, which means the response is now handled pretty much as responses to plain INVITEs are. We should probably handle the distinction better, but in my simple tests it seemed to be working for now. There will be room for improvement, assuming you guys play with this and give me feedback of course.

[chest3x]

The segmentation fault is fixed. However now, insted of a segfault, I am getting "Not a recording session, can't refresh" message. Maybe it is supposed to work that way, I don't know.

My behavior on client is:

1. click "Record" button, fill the name, press OK
2. click "Stop" red button on top of the video
3. click "Record" button, fill the name again, press OK - this returns the message "Not a recording session, can't refresh" on client

It appears to me like, Janus is trying to use ICE restart after second attempt to record the video on the same session.

These are the last words from the server:

File is 1206663 bytes: rec-5325994183735943-video.mjr
Closed video recording rec-5325994183735943-video.mjr
[WARN] [3574993280390131] Didn't receive audio for more than  second...
[WARN] [3574993280390131] Didn't receive video for more than a second...
[WARN] [3574993280390131] Negotiation update, checking what changed...
[WARN] [3574993280390131] ICE restart detected
[WARN] [3574993280390131] Restarting ICE...
[ERR] [plugins/janus_recordplay.c:janus_recordplay_handler:1119] Not a recording session, can't refresh

I am using the clean, non-modified version of Janus (ice-restart branch)

[lminiero]

Yep, I think you're probably right. I'll look into this.

[lminiero]

Can't replicate this, it works just fine for me...

[chest3x]

I am experiencing this on recent Firefox, Windows 10.

On Chrome it works, however it is returning this warning after stopping the recording:

[WARN] Got a stop but missing recorder/recording! No .nfo file generated...

[lminiero]

I am experiencing this on recent Firefox, Windows 10.

Just checked and the problem are not ICE restarts, here, but the fact that the PeerConnection is not recognized as gone, with Firefox, and so it's not closed after you hit stop. To make this clearer, we do hangup the PeerConnection on the JavaScript side, but Firefox does not send a DTLS alert, and so Janus is not made aware the PeerConnection is gone. The same issue happens with master: you try to record again, and nothing happens, because Janus thinks it's a renegotiation and it ignores it. I think the same thing happened in the past. I'll have to think of a way to make sure PeerConnections are closed on both ends for cases like this.

On Chrome it works, however it is returning this warning after stopping the recording:

[WARN] Got a stop but missing recorder/recording! No .nfo file generated...

Probably a bug somewhere, as I noticed that too but the .nfo files are definitely there and playable. My guess is that when it reaches that code, the recording object has already been removed (it can happen after different triggers) and so it thinks it never existed while it really did.

[mirkobrankovic]

I have tested this PR against VideoRoom plugin, and seems fine by me.
WebRTC callback for PC.oniceconnectionstatechange() is a bit slower, takes few seconds to receive failed (or i did something wrong in my test case - disable/enable wifi), but after that takes about one second or less to get streams back up :D

@lminiero, great work, I would love to see this merged, cheers

[lminiero]

@mirkobrankovic thanks for testing! I'll actually have to update this PR, as it was written before I refactored the SDP utils and so will need some work. Merging this will need to wait until we merge the reference counters, though, as that's what it's based on.

[lminiero]

Closing as this has diverged too much from the refcount branch: I've integrated all the changes this introduced in a new PR with a wider purpose, #1099, so please refer to that if you were interested in this effort.

[fanlion]

For subscribers, just send a configure with a refresh:true property on the subscriber handle to trigger an ICE restart. As in the Streaming plugin, it will result in a new offer from Janus, that you can answer the usual way.

feeds[1].send({message: {request: "configure", refresh: true}});

when i do this, janus server didnot send a new offer to subscriber handler,

finally, i change refresh to restart, it work for me

[harshald28]

For subscribers, just send a configure with a refresh:true property on the subscriber handle to trigger an ICE restart. As in the Streaming plugin, it will result in a new offer from Janus, that you can answer the usual way.

feeds[1].send({message: {request: "configure", refresh: true}});

when i do this, janus server didnot send a new offer to subscriber handler,

finally, i change refresh to restart, it work for me

The "refresh" attribute renamed it to "update".
7ac3108

Also, As per Lorenzo's comment in 1099
"restart: true forces an ICE restart, update: true simply indicates a renegotiation."