H264 `packetizationMode` array in `RtpCodecCapability`

[ibc]

By implementing this, I've found that having packetizationMode as an array of integers in the capability codec of H264 is not comfortable at all.

Currently, a browser can expose its H264 capabilities with a single codec capability entry containing packetizationMode: [ 0, 1 ] but the fact is that, depending which value is chosen, that is a different codec+configuration. However, the browser may want to call rtpSender.send() with both codec_A (H264 + packetizationMode: 0) and codec_B (H264 + packetizationMode: 1) so it needs TWO different preferredPayloadTypes for them, rather than a single one.

[ibc]

To be clear: when Firefox generates an SDP with H264 it has two H264 entries, with different payload types and different packetization-mode. Those are two different codecs so, why should the ORTC RtpSender.getCapabilities('video') provide a single H264 entry with an array of supported packetizationMode values and a single preferredPayloadtype?

[aboba]

@ibc The only scenario that comes to mind in which a browser may want to call rtpSender.send() with both codec_A (H264 + packetizationMode: 0) and codec_B (H264 + packetizationMode: 1) would be a conference with a legacy system (supporting only packetization-mode 0) as well as WebRTC browser clients (supporting packetization-mode 1, which is mandatory-to-implement). To accommodate both, the sender simulcasts both modes 0 and 1, with an SFU in-between to decide which stream to send to a given client.

However, the above scenario is not very mainstream. In most cases, I would expect the "common capabilities" algorithm to compute the intersection of the packetization-mode arrays and if there is more than one mode available, utilizes a preference (probably for packetization-mode 1) to select the mode to use, so that only one preferredPayloadType is needed.

[robin-raymond]

The reason to not create multiple capabilities entries is because each packetization mode would consume a preferred payload type and payload types are limited. As an array, common scenarios (which only require 0, 1, or 2) would pick their mode and be done as a mutual agreed value which is going to be the most common scenario.

In the rarer simulcast scenarios (or where you must advertise them as separate entries) you can manually assign your own PT value and split these as separate codes with packetization mode if you so choose for each PT. You are not required to use the preferred payload type and can assign your own. Thus if you need on stream with PT 98 with mode 0 and another with PT 99 and mode 1, you are allowed to do that yourself.

So to avoid consuming rare PTs for the common cases and since we can support the other scenario too I suggest nothing should be done and marked as "won't fix". Or maybe clarification and illustrate how this would be handled using the existing API.

[aboba]

I believe this is addressed by PR #634