MCU Audio Only Mode

[zaleschoenborn]

Is your feature request related to a problem? Please describe.
Audio only MCU

There is no way to merge many published audio only mediastreams into one audio mediastream.

Audio only MCU is important as it should scale to much higher numbers than a fixed function video + audio mcu

Describe the solution you'd like
Sum all published audio mediastreams in a room into one combined output audio mediastream

Each published mediatrack channel in a mediastream should be summed with the other respective published audio mediastream track channels. (i.e. The output mediastream should have the same number of audio mediatrack channels as the input mediastream so the SFU preserves the channel count to achieve mono, stereo, 5 channel, 7 channel, etc).

Describe alternatives you've considered
Building a client that can sum the audio mediastreams using a audio only SFU is expensive and harder to scale to a large number of published audio mediastreams.

Additional context
Add any other context or screenshots about the feature request here.

[peterzanetti]

Audio only MCU is a very good idea. This is the ideal way to do audio while keeping video as SFU. We would adopt this if it were available.

[zaleschoenborn]

More detailed implementation notes...

Please use the FFmeg amix filter in a plug in to do the audio merge.

You will need to use variables in the FFmpeg AMIx filter where the following input variables dynamically update

N = current number of published stream inputs in the room

AudioStreams is an array of N audio streams

The normalize variable in the AMIX filter enforces that the volume of any input is divided by the number of streams (N) in the room. you should allow the user to define if they want the normalization or not for their application.

Two modes this would be most useful

Scenario 1 (traditional MCU)

MCU plug-in (room Id, expose AMIX normalize variable)

Questions...
How does this scale with many rooms in a cluster. What happens when a specific server hits a maximum load? How will an application know? Lots of issue and concerns here but open to your teams ideas.

Scenario 2 Simpler Audio merge plug in for scale that doesn't require room
Logic (ie does not require knowledge of any other servers)

Create a new Audio merge plug that can be installed be origin in a cluster that doesn't require a room input but still expose Ami normalize variable.

Once the Audio merge plug becomes active it would need to automatically generate a unique merged stream ID and create a way for the application to discover it (listener/event?) and playback the created merged streamID

This scales better because an application can

1. stand up n origin servers to start and discover all of the merged audio streams to playback

2. if any origin becomes to loaded a new origin can be automatically added in the cluster and a new merged audio stream Id event is fired so it can be discovered and played by the application.

The scale pattern could continues to handle merging an unlimited number of input stream.

Look forward to this feature!

[peterzanetti]

As above poster mentions, Origin/Edge and cluster should be considered from the start and part of the design.

And maybe give some thought to allowing integration of a SIP call into the Audio merge. Open the possibility of a third part conference call provider like Iotum that could bridge a conference call with the MCU stream.

I would love to be able to ditch my current MCU provider and run my own on Ant Media Server.