Problem with setting "mixer" rtpLevelRelayType for RTPChannels

[romix]

I think recent changes to Jitsi-videobridge introduced a small bug.

Constructor of RTPChannel performs this call even before the XML attributes of the channel are parsed:

https://github.com/jitsi/jitsi-videobridge/blob/master/src/org/jitsi/videobridge/RtpChannel.java#L205

 if (RTPLevelRelayType.MIXER.equals(getRTPLevelRelayType()))

As a result, getRTPLevelRelayType sets the type to a default, which is a "translator". When attributes of the channel's XML description are parsed later and the attribute rtp-level-relay-type="mixer" is discovered later in Videobridge.handleColibriConferenceIQ, the code tries to set the type to "mixer". But it does not work, because the logic in RtpChannel.setRTPLevelRelayType allows to set a type only if it is not set set or it is the same. Since each RTPChannel has now "translator" as an initially set type, it is impossible to set "mixer" at all, as far as I understand.

This seems to be a regression from previous versions of the video bridge. I guess it should be still possible to set a "mixer" mode during conference establishment.

[romix]

Thanks for working on this.

Actually this change does not completely fix the problem.
The good thing is that it now allows setting "mixer" as a type (see below a response from the videobridge)

But it still forgets to add "source" information to the audio channel inside the colibri payload being sent out. At the same time, the video channel information contains now the "source" information. Earlier versions of jitsi-videobridge did it the other way around: an audio channel would contain a "source" element and a video channel would not contain it.

Could you check what is still wrong? May be "source" is simply being added to the wrong channel?

Here is a trace to help you with fixing a problem:

This is what we send to jitsi-videobridge:

<iq type="get" id="offer-hco5t58c8utgb51o18-1404454226887" from="jitsi.videobridge.conf180gq80hf1yt2ki8iw@localhost/jitsi.videobridge.conf180gq80hf1yt2ki8iw" to="jitsi-videobridge.mydomain.localhost">
    <conference xmlns="http://jitsi.org/protocol/colibri">
        <content name="audio">
             <channel initiator="true" expire="15" rtp-level-relay-type="mixer" endpoint="ep1"/>
        </content>
        <content name="video">
             <channel initiator="true" expire="15" last-n="1" endpoint="ep1"/>
        </content>
    </conference>
</iq>

Then jitsi-videobridge produces this response:

<iq id="offer-hco5t58c8utgb51o18-1404454226887" to="jitsi.videobridge.conf180gq80hf1yt2ki8iw@localhost/jitsi.videobridge.conf180gq80hf1yt2ki8iw" from="jitsi-videobridge.mydomain.localhost" type="result">
<conference xmlns="http://jitsi.org/protocol/colibri" id="5e8fa5e1ea190002">
  <content name="audio">
    <channel direction="recvonly" endpoint="ep1" expire="15" id="10a8a5d3dfaa2efb" initiator="true" rtp-level-relay-type="mixer">
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" pwd="2i28jav4rcpmv6q25jsd8bop2u" ufrag="44d8v">
        <fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0" hash="sha-1">EC:00:17:0E:F4:D7:EF:2F:30:E2:7D:D5:18:A6:5D:42:B3:C5:22:B1</fingerprint>
        <candidate component="1" foundation="1" generation="0" id="5e8fa5e1ea190002364767f74b15362541ea91302f49f57f" ip="fe80:0:0:0:288:65ff:fe38:df4" network="0" port="10004" priority="2130706431" protocol="udp" type="host"/>
        <candidate component="1" foundation="2" generation="0" id="5e8fa5e1ea190002364767f74b15362541ea913022bc47be" ip="192.168.178.21" network="0" port="10004" priority="2130706431" protocol="udp" type="host"/>
        <candidate component="1" foundation="3" generation="0" id="5e8fa5e1ea190002364767f74b15362541ea91301844086e" ip="212.202.121.5" network="0" port="10004" priority="1677724415" protocol="udp" rel-addr="192.168.178.21" rel-port="10004" type="srflx"/>
        <candidate component="2" foundation="1" generation="0" id="5e8fa5e1ea190002364767f74b15362541ea91302c182ac8" ip="fe80:0:0:0:288:65ff:fe38:df4" network="0" port="10005" priority="2130706430" protocol="udp" type="host"/>
        <candidate component="2" foundation="2" generation="0" id="5e8fa5e1ea190002364767f74b15362541ea9130def5869" ip="192.168.178.21" network="0" port="10005" priority="2130706430" protocol="udp" type="host"/>
        <candidate component="2" foundation="3" generation="0" id="5e8fa5e1ea190002364767f74b15362541ea91301683f516" ip="212.202.121.5" network="0" port="10005" priority="1677724414" protocol="udp" rel-addr="192.168.178.21" rel-port="10005" type="srflx"/>
      </transport>
    </channel>
  </content>
  <content name="video">
    <channel endpoint="ep1" expire="15" id="54bf60e7c1bfad37" initiator="true" last-n="1" rtp-level-relay-type="translator">
      <source xmlns="urn:xmpp:jingle:apps:rtp:ssma:0" ssrc="3251051638"/>
      <transport xmlns="urn:xmpp:jingle:transports:ice-udp:1" pwd="2s7cmtn8h8uj6mn2uhj9nojiij" ufrag="f5bg9">
        <fingerprint xmlns="urn:xmpp:jingle:apps:dtls:0" hash="sha-1">69:1F:43:98:E9:0E:0B:5F:49:80:01:C1:49:F1:C9:67:18:5B:01:D7</fingerprint>
        <candidate component="1" foundation="1" generation="0" id="5e8fa5e1ea190002434de43f19e135435c3a439107c212e8d" ip="fe80:0:0:0:288:65ff:fe38:df4" network="0" port="10006" priority="2130706431" protocol="udp" type="host"/>
        <candidate component="1" foundation="2" generation="0" id="5e8fa5e1ea190002434de43f19e135435c3a4391050e1412a" ip="192.168.178.21" network="0" port="10006" priority="2130706431" protocol="udp" type="host"/>
        <candidate component="1" foundation="3" generation="0" id="5e8fa5e1ea190002434de43f19e135435c3a4391012bc631e" ip="212.202.121.5" network="0" port="10006" priority="1677724415" protocol="udp" rel-addr="192.168.178.21" rel-port="10006" type="srflx"/>
        <candidate component="2" foundation="1" generation="0" id="5e8fa5e1ea190002434de43f19e135435c3a4391040ba2d58" ip="fe80:0:0:0:288:65ff:fe38:df4" network="0" port="10007" priority="2130706430" protocol="udp" type="host"/>
        <candidate component="2" foundation="2" generation="0" id="5e8fa5e1ea190002434de43f19e135435c3a439104e12c815" ip="192.168.178.21" network="0" port="10007" priority="2130706430" protocol="udp" type="host"/>
        <candidate component="2" foundation="3" generation="0" id="5e8fa5e1ea190002434de43f19e135435c3a4391057beff2d" ip="212.202.121.5" network="0" port="10007" priority="1677724414" protocol="udp" rel-addr="192.168.178.21" rel-port="10007" type="srflx"/>
      </transport>
    </channel>
  </content>
</conference>
</iq>

[romix]

This looks much better now. We get audio SSRCs in mixed mode.

But when we test end-to-end with 4 participants and mixed mode, we observe strange effects:

1. Sometimes, the sound would appear only 20-30 seconds after a participant has joined the conf and after video has arrived. This is very strange. It is a bit similar to a situation which sometimes happens in video apps, where you wait for a next key frame. But we have never seen this with audio till now ;-(

2. Some participants can hear the incoming audio stream (typically those who use OS X/Chrome). Some others (typically, Ubuntu/Chrome) cannot hear anything. But we are not quite sure that it is related to OS/Chrome combination.

The relay mode seems to work just fine with audio and video.

One month ago or even a bit later, we had no problems with mixed.

We are trying to figure out the reason why some people can hear the mixed audio and some others - not. We are not sure, if it is a videobridge problem or a problem with our Web Client (we use our custom Web Client, not JitMeet). I'll keep you posted.

BTW, these audio problems happen with and without last-N, as it looks like.

One more thing, which is most likely a totally different issue:
lastN and activeSpeaker events seem to work, but a bit strange....

For example, lastN is sent out rather rare. More over, when a new participant joins, one could think that he should get a lastN event (and may be active speaker as well) so that he knows which video streams he receives. But it does not happen. The new participant receives streams according to last-N setting, but does not receive the lastN event ....

Active speaker events sometimes work as expected, sometimes they are sent rather rare.

These are just our observations about this new activeSpeaker/lastN functionality....

[lyubomir]

Thank you for the report. However, I'll have to ask you to discuss that on the dev mailing list because (1) the subject of this issue is about the mixer rtp-level-relay-type attribute value being ignored and (2) the latest commnet is like the third and fourth problems reported in this issue and the second and third problems unrelated to the subject of this issue.

[romix]

@lyubomir Point taken. And I even stated in the comment that those lastN and active speaker related things are orthogonal to the original issue.

However, do you think that these strange mixed audio issues (some participants can hear it, some others not) may be related to the recent changes related to handling of mixed and translator rtp level relay types? After all, before those changes we tested and everything seemed to work fine in mixed mode.