Dominant Speaker Event

[SteveMcFarlin]

Implementation of Dominant Speaker Identification for Multipoint Videoconferencing by Ilana Volfin and Israel Cohen. This implementation uses the RTP Audio Level extension from RFC-6464 for the input signal. This has been ported from DominantSpeakerIdentification.java in Jitsi

[ibc]

Amazing, thanks. First review round done.

[nazar-pc]

It would be great to include corresponding changes on Rust side as well 🙏

[SteveMcFarlin]

It would be great to include corresponding changes on Rust side as well 🙏

I took a look at the Rust implementation. I am slightly confused. In the implementation for audio_level_observer.rs it appears the RtpObserver is also implemented in that file. In any case I thought perhaps I could "hack" it if the code was not much. Basically by following the core audio_level_observer.rs implementation. Well I thought wrong. I have only ever played with Rust, so I don't think I can help here.

[nazar-pc]

In the implementation for audio_level_observer.rs it appears the RtpObserver is also implemented in that file.

It is not implementation of RtpObserver itself, it just implements RtpObserver trait for AudioLevelObserver struct. Think of it as implementing an interface for a class in other languages.

[SteveMcFarlin]

@nazar-pc Please take a look at the rust implementation. Thanks!

[nazar-pc]

Rust version looks good, thank you!

[ibc]

CI is failing due to this:

../include/RTC/ActiveSpeakerObserver.hpp:6:10: fatal error: json.hpp: No such file or directory
    6 | #include <json.hpp>
      |          ^~~~~~~~~~

It must be #include <nlohmann/json.hpp>.

[ibc]

Thanks, looking great. Waiting for CI to complete.

BTW could you please contribute to the mediasoup-website project with a PR to document the new API?

[ibc]

Thanks a lot for this, guys. I'll wait for the documentation PR before merging. Just let me know if you have any problem or question writing it. There is "gulp live" task in mediasoup-website project to run the web locally.

[ibc]

Released in mediasoup v3.8.0 :)

Thanks a lot!

[SteveMcFarlin]

Thank you for the great review process!

[msach22]

@SteveMcFarlin nice contribution 😄

[mk19831002]

Hi,
did you write the documentation for that, I didn't find it in API documentation.

[ibc]

Somehow the website is not up to date in the web server, but yes, he wrote docs. I'll check and republish later today.

[d2e1]

Hello guys,
I was testing the dominant speaker feature and a situation I came across prompted me to review the code. I also reviewed the Jitsi code and the article the algorithm is based on. As far as I can see, the following piece of code is added differently in Jitsi:

Mediasoup code:
https://github.com/versatica/mediasoup/blob/v3/worker/src/RTC/ActiveSpeakerObserver.cpp#L313

Jitsi code:
https://github.com/jitsi/jitsi-utils/blob/master/src/main/java/org/jitsi/utils/dsi/DominantSpeakerIdentification.java#L715

Related part of the article (Dominant Speaker Identification for Multipoint Videoconferencing by Ilana Volfin and Israel Cohen)

In the article and Jitsi code, the dominant speaker is included in the dividing part. I don't know the other parts of the code well, I wonder if a change is needed here?

Link for the open discussion : https://mediasoup.discourse.group/t/question-regarding-active-speaker-observer-implementation/3227

[punarinta]

Hi @SteveMcFarlin,

Could you please advise me what is the reason I'm not getting any events from ActiveSpeakerObserver instance?

I've initialized the top-level code like

const observer = await router.createActiveSpeakerObserver({interval: 500})

observer.on('dominantspeaker', (payload) => {
	console.log('event payload', payload)
})

For debugging I've added logs after lines 19 and 20 in https://github.com/versatica/mediasoup/blob/v3/node/lib/ActiveSpeakerObserver.js and I see that handleWorkerNotifications is called, but this.channel.on never fires its callback.

Both audio and video is sent and received correctly between the participating peers, so audio data definitely flows the way it should.

UPD: discussion moved to https://mediasoup.discourse.group/t/activespeakerobserver-does-not-emit-events/3621

[ibc]

Please let use the forum for questions.