determining common capabilities

[fippo]

Trying to fix my remaining items for h264 interop between Chrome/Firefox and Edge I ran into an issue with determining the intersection of codec parameters and rtcpFeedback. It would be useful to at least have some high-level text about matching local and remote capabilities in the spec.

Some code lives here. The ORTC demo on Microsofts TestDrive also has a function filterCodecParams which does a similar thing.

After I determine that a codec is common (same name, clockrate and number of channels), I need to determine what rtcp feedback and parameters can be used.

rtcp-fb is easy, chrome sends this:

• ccm fir
• nack
• nack pli
• goog-remb
• transport-cc

Since Edge only supports nack pli and goog-remb, only those two should be in the answer (I am not sure if there is a distinction between 'capabilities in the answer' and 'common capabilities').
This is also described in RFC 4585:

   When used in conjunction with the offer/answer model [8], the offerer
   MAY present a set of these AVPF attributes to its peer.  The answerer
   MUST remove all attributes it does not understand as well as those it
   does not support in general or does not wish to use in this
   particular media session.

The parameters are something where I need advice. When Firefox offers, the profile-level-id is

• profile-level-id=42e01f

whereas Edge has this capability

• profile-level-id=42C02A

How do I deal with this? A filter operation as I do for rtcp-fb is not right.

Same problem for audio (so its not just a matter of h264 profile levels), Chrome sends
a=fmtp:111 minptime=10; useinbandfec=1
which currently gets passed into Edge's RTPSender.send (without issues, I assume it ignores them).

Should I rather put the 'left' (local) parameters into the answer and the offerer (browser engine) is responsible for figuring out the right thing to do?

This smells a lot like the offer-answer rules from RFC 3264:

   The interpretation of fmtp parameters in an offer depends on the
   parameters.  In many cases, those parameters describe specific
   configurations of the media format, and should therefore be processed
   as the media format value itself would be.  This means that the same
   fmtp parameters with the same values MUST be present in the answer if
   the media format they describe is present in the answer.  Other fmtp
   parameters are more like parameters, for which it is perfectly
   acceptable for each agent to use different values.  In that case, the
   answer MAY contain fmtp parameters, and those MAY have the same
   values as those in the offer, or they MAY be different.  SDP
   extensions that define new parameters SHOULD specify the proper
   interpretation in offer/answer.

the tl;dr of this is 'it depends' i guess

[ibc]

The parameters are something where I need advice. When Firefox offers, the profile-level-id is

• profile-level-id=42e01f
  whereas Edge has this capability
• profile-level-id=42C02A

How do I deal with this?

Easy. From RFC 6184 section 8.1:

profile-level-id:

     A base16 [7] (hexadecimal) representation of the following
     three bytes in the sequence parameter set NAL unit is specified
     in [1]: 1) profile_idc, 2) a byte herein referred to as
     profile-iop, composed of the values of constraint_set0_flag,
     constraint_set1_flag, constraint_set2_flag,
     constraint_set3_flag, constraint_set4_flag,
     constraint_set5_flag, and reserved_zero_2bits in bit-
     significance order, starting from the most-significant bit, and
     3) level_idc.  Note that reserved_zero_2bits is required to be
     equal to 0 in [1], but other values for it may be specified in
     the future by ITU-T or ISO/IEC.

     The profile-level-id parameter indicates the default sub-
     profile (i.e., the subset of coding tools that may have been
     used to generate the stream or that the receiver supports) and
     the default level of the stream or the receiver supports.

     The default sub-profile is indicated collectively by the
     profile_idc byte and some fields in the profile-iop byte.
     Depending on the values of the fields in the profile-iop byte,
     the default sub-profile may be the set of coding tools
     supported by one profile, or a common subset of coding tools of
     multiple profiles, as specified in Section 7.4.2.1.1 of [1].
     The default level is indicated by the level_idc byte, and, when
     profile_idc is equal to 66, 77, or 88 (the Baseline, Main, or
     Extended profile) and level_idc is equal to 11, additionally by
     bit 4 (constraint_set3_flag) of the profile-iop byte.  When
     profile_idc is equal to 66, 77, or 88 (the Baseline, Main, or
     Extended profile), level_idc is equal to 11, and bit 4
     (constraint_set3_flag) of the profile-iop byte is equal to 1,
     the default level is Level 1b.

     Table 5 lists all profiles defined in Annex A of [1] and, for
     each of the profiles, the possible combinations of profile_idc
     and profile-iop that represent the same sub-profile.

buff...

[ibc]

Somehow related: #567

[rshpount]

On Wed, Jun 22, 2016 at 12:34 PM, Philipp Hancke notifications@github.com
wrote:

Same problem for audio (so its not just a matter of h264 profile levels),
Chrome sends

a=fmtp:111 minptime=10; useinbandfec=1

OPUS parameters are not negotiated. They are independent in each
direction. There is no way to see which parameters are supported and which
are ignored. So, Chrome is saying that it does not want OPUS with ptime
less then 10 ms and asking to use in band fec when sending audio to it. You
should get the codec parameters supported by Edge and send them to Chrome
to indicate Edge preferences in opposite direction.

---

Roman Shpount

[fippo]

Do I have to accept the grim reality that there is no generic way to do this but I need to teach my JS all kinds of weird information about h264 and opus parameters? :-/

[ibc]

Yes.

[aboba]

@fippo @robin-raymond With respect to codec-specific parameters, most settings fit into one of the following buckets:

a. A boolean capability of the receiver that can be set as a parameter on the sender. Examples: useDtx and useInbandFec in Opus. If the receiver indicates useDtx = true, then it would prefer the sender to use DTX, so that useDtx = true can be set in sender parameters. Similarly, if the receiver indicates useInbandFec = true, then it would prefer the sender to use inband FEC, so that useInbandFec = true can be set in sender parameters. Note that useDtx and useInbandFec is not a sender capability because it is assumed that a sender will support it (this isn't always true, but if it isn't the sender should ignore the setting).

b. A numeric capability of the receiver that can be set as a parameter on the sender. maxBitrate, maxFr and maxFs are examples of this.

c. Weird cases. I think that profileLevelId may be one of these. As I read RFC 6184, when level-asymmetry = 0 or not provided (e.g. Chrome), the levels sent and received need to be the same. So in this case, you can take a receiver capability of profile-level-id=42e01f and set this on a sender (Edge) that has a capability of profile-level-id=42C02A but having Edge answer with profile-level-id=42C02A would not work because the levels need to be symmetric - you need to take the minimum of the capabilities and have Edge answer with profile-level-id=42e01f. Since with FF level-asymmetry=1, the situation may be different there (does FF tolerate an Answer of profile-level-id=42C02A?)

[aboba]

@fippo Looking at the code, I would suggest the following approach to determining sender parameters (the intersection of the local Sender capabilities and the remote Receiver capabilities).

RTCRtpCodecCapability. If the local codecs[i].name and codecs[i].clockRate match values within the remote capabilities, then we have a codec match, but need to figure out the values of other attributes for codecs[i]:

a. preferredPayloadType: this is the remote codecs[i].preferredPayloadType value.

b. ptime: this is set to the sender value, clamped by the receiver maxptime.

c. numChannels: this is the smaller of the local and remote values.

d. rtcpFeedback: this is the intersection of the supported values of the local sender and remote receiver for a given codec.

[fippo]

I ran into another issue here when it came to RTX. The current code I have in adapter broken when it got something along the lines of:

a=rtpmap:107 H264/90000
a=rtpmap:100 VP8/90000
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=107
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=100

Since I was merely checking that the left (local) and right (remote) code name matched I ended up putting rtx with the wrong apt in the common set (which didn't work).
RTX needs special handling here. If the codec name is rtx, there must be a local rtx codec with an apt that is associated with a matching codec.

sigh

[aboba]

Still outstanding:
a. More advice about common codec parameters (for codecs other than H.264/AVC and Opus).
b. Determination of common FEC mechanisms (not in the cited example function)
c. Incorporation of common FEC mechanisms within RTCRtpSend/ReceiveParameters.codecs[]
d. Advice on ptime