flexible FEC for video

[robin-raymond]

Doesn't appear to be any way to describe flexible FEC in ORT specification...

https://tools.ietf.org/html/draft-ietf-rtcweb-fec-01#section-5.1

5.1.  Recommended Mechanism

   For video content, use of a separate FEC stream with the RTP payload
   format described in [I-D.ietf-payload-flexible-fec-scheme] is
   RECOMMENDED.  The receiver can demultiplex the incoming FEC stream by
   SSRC and correlate it with the primary stream via the ssrc-group
   mechanism.

   Note that this only allows the FEC stream to protect a single primary
   stream.  Support for protecting multiple primary streams with a
   single FEC stream is complicated by WebRTC's 1-m-line-per-stream
   policy and requires further study.

5.2.  Negotiating Support

   To offer support for a separate FEC stream, the offerer MUST offer
   one of the formats described in
   [I-D.ietf-payload-flexible-fec-scheme], Section 5.1, as well as a
   ssrc-group with "FEC-FR" semantics as described in [RFC5956],
   Section 4.3.

   Answerers can reject the use of FEC by not including FEC payloads in
   the answer.

https://tools.ietf.org/html/draft-ietf-payload-flexible-fec-scheme-00#section-5.1

5.  Payload Format Parameters

   This section provides the media subtype registration for the non-
   interleaved and interleaved parity FEC.  The parameters that are
   required to configure the FEC encoding and decoding operations are
   also defined in this section.

7.1.  Example SDP for 1-D Parity FEC Protection

   In this example, we have one source video stream (ssrc:1234) and one
   FEC repair stream (ssrc:2345).  We form one FEC group with the
   "a=ssrc-group:FEC-FR 1234 2345" line.  The source and repair streams
   are multiplexed on different SSRCs.  The repair window is set to 200
   ms.

        v=0
        o=ali 1122334455 1122334466 IN IP4 fec.example.com
        s=1-D Interleaved Parity FEC Example
        t=0 0
        m=video 30000 RTP/AVP 100 110
        c=IN IP4 233.252.0.1/127
        a=rtpmap:100 MP2T/90000
        a=rtpmap:110 interleaved-parityfec/90000
        a=fmtp:110 L:5; D:10; ToP:0; repair-window:200000
        a=ssrc:1234
        a=ssrc:2345
        a=ssrc-group:FEC-FR 1234 2345

7.2.  Example SDP for 2-D Parity FEC Protection

   In this example, we have one source video stream (ssrc:1234) and two
   FEC repair streams (ssrc:2345 and ssrc:3456).  We form one FEC group
   with the "a=ssrc-group:FEC-FR 1234 2345 3456" line.  The source and
   repair streams are multiplexed on different SSRCs.  The repair window
   is set to 200 ms.

        v=0
        o=ali 1122334455 1122334466 IN IP4 fec.example.com
        s=2-D Parity FEC Example
        t=0 0
        m=video 30000 RTP/AVP 100 110 111
        c=IN IP4 233.252.0.1/127
        a=rtpmap:100 MP2T/90000
        a=rtpmap:110 interleaved-parityfec/90000
        a=fmtp:110 L:5; D:10; ToP:2; repair-window:200000
        a=rtpmap:111 non-interleaved-parityfec/90000
        a=fmtp:111 L:5; D:10; ToP:2; repair-window:200000
        a=ssrc:1234
        a=ssrc:2345
        a=ssrc:3456
        a=ssrc-group:FEC-FR 1234 2345 3456

   Note that the sender might be generating two repair flows carrying
   non-interleaved and interleaved FEC packets, however the receiver
   might be interested only in the interleaved FEC packets.  The
   receiver can identify the repair flow carrying the desired repair
   data by checking the payload types associated with each repair flow
   described in the SDP description.

https://tools.ietf.org/html/rfc5956#section-4.3

[RFC5576] defines an SDP media-level attribute, called "ssrc-group",
   for grouping the RTP streams that are SSRC multiplexed and carried in
   the same RTP session.  The grouping is based on the Synchronization
   Source (SSRC) identifiers.  Since SSRC-multiplexed RTP streams are
   defined in the same "m" line, the "group" attribute cannot be used.

   This section specifies how FEC is applied to source and repair flows
   for SSRC-multiplexed streams using the "ssrc-group" attribute
   [RFC5576].  This section also specifies how the additivity of the
   repair flows is expressed for the SSRC-multiplexed streams.

   The semantics of "FEC-FR" for the "ssrc-group" attribute are the same
   as those defined for the "group" attribute, except that the SSRC
   identifiers are used to designate the FEC grouping associations:
   a=ssrc-group:FEC-FR *(SP ssrc-id) [RFC5576].

   The SSRC identifiers for the RTP streams that are carried in the same
   RTP session MUST be unique per [RFC3550].  However, the SSRC
   identifiers are not guaranteed to be unique among different RTP
   sessions.  Thus, the "ssrc-group" attribute MUST only be used at the
   media level [RFC5576].

   Let us consider the following scenario where there are two source
   flows (e.g., one video and one audio) and a single repair flow that
   protects only one of the source flows (e.g., video).  Suppose that
   all these flows are separate RTP streams that are SSRC multiplexed in
   the same RTP session.

                  SOURCE FLOWS             | FEC FRAMEWORK INSTANCE #1
                  S5: Source Flow |--------| R8: Repair Flow
                  S6: Source Flow

    Figure 4: Example scenario with one FEC Framework instance where a
         single repair flow protects only one of the source flows

   The following SDP describes the scenario sketched in Figure 4.

        v=0
        o=ali 1122334455 1122334466 IN IP4 fec.example.com
        s=FEC Grouping Semantics for SSRC Multiplexing
        t=0 0
        m=video 30000 RTP/AVP 100 101 110
        c=IN IP4 233.252.0.1/127
        a=rtpmap:100 JPEG/90000
        a=rtpmap:101 L16/32000/2
        a=rtpmap:110 1d-interleaved-parityfec/90000
        a=fmtp:110 L=5; D=10; repair-window=200000
        a=ssrc:1000 cname:fec@example.com
        a=ssrc:1010 cname:fec@example.com
        a=ssrc:2110 cname:fec@example.com
        a=ssrc-group:FEC-FR 1000 2110
        a=mid:Group1

   Note that in actual use, SSRC values, which are random 32-bit
   numbers, may be much larger than the ones shown in this example.
   Also, note that before receiving an RTP packet for each stream, the
   receiver cannot know which SSRC identifier is associated with which
   payload type.

   The additivity of the repair flows is handled in the same way as
   described in Section 4.2.  In other words, the repair flows that are
   included in an "a=ssrc-group" line MUST be additive.  Repair flows
   that are not additive MUST be indicated in separate "a=ssrc-group"
   lines.

[robin-raymond]

For red codec to work in combination with ulpfec, i.e. "red+ulpfec" mechanism, red codec needs to define the encapsulation of ulpfec.

https://tools.ietf.org/html/rfc5109#section-14.2

  For example:

      m=audio 12345 RTP/AVP 121 0 5 100
      a=rtpmap:121 red/8000/1
      a=rtpmap:100 ulpfec/8000
      a=fmtp:121 0/5/100

As such, we need RTCRtpCodecParameters.parameters to define the "fmtp", which is basically a list of packets encapsulated within the red codec.

For for the red, I propose a sender and receiver parameter inside RTCRtpCodecParameters.parameters that contains:
sequence<payloadtype> payloadTypes - the list of payload types encapsulated within the red packet [as defined within https://tools.ietf.org/html/rfc5109#section-14.2 fmtp]

So for example let's say red was payload 100 and ulpfec was payload 101, the red+ulpfec within chromium as its currently used would be defined as:

{
"name": "ulpfec",
"payloadType": 101,
...
},
{
"name": "red",
"payloadType": 100,
...
"parameters" : {
 "payloadTypes": [
      101
    ]
  }
}

In the above example, red would re-use the same SSRC as the main stream with this mechanism. From https://tools.ietf.org/html/rfc5109#section-14.2 :

   Because the FEC data (including the ULP header) is sent in the same
   packets as the protected payload, the FEC data is associated with the
   protected payload by being bundled in the same stream.

There's only one catch though, as discovered with chromium, the video packets are often too large to include both main codec encoding and fec packet inside the same red packet so it's possible the ulpfec is exclusive inside the red packet and the main codec are interleaved within the same SSRC.

[robin-raymond]

https://tools.ietf.org/html/draft-ietf-payload-flexible-fec-scheme-01

   Required parameters:

   o  rate: The RTP timestamp (clock) rate.  The rate SHALL be larger
      than 1000 Hz to provide sufficient resolution to RTCP operations.
      However, it is RECOMMENDED to select the rate that matches the
      rate of the protected source RTP stream.

   o  repair-window: The time that spans the source packets and the
      corresponding repair packets.  The size of the repair window is
      specified in microseconds.

   Optional parameters:

   o  L: indicates the number of columns of the source block that are
      protected by this FEC block and it applies to all the source
      SSRCs.  L is a positive integer.

   o  D: indicates the number of rows of the source block that are
      protected by this FEC block and it applies to all the source
      SSRCs.  D is a positive integer.

   o  ToP: indicates the type of protection applied by the sender: 0 for
      1-D interleaved FEC protection, 1 for 1-D non-interleaved FEC
      protection, and 2 for 2-D parity FEC protection.  The ToP value of
      3 is reserved for future uses.

7.1.  Example SDP for Flexible FEC Protection with in-band SSRC mapping

   In this example, we have one source video stream and one FEC repair
   stream.  The source and repair streams are multiplexed on different
   SSRCs.  The repair window is set to 200 ms.

        v=0
        o=mo 1122334455 1122334466 IN IP4 fec.example.com
        s=FlexFEC minimal SDP signalling Example
        t=0 0
        m=video 30000 RTP/AVP 96 98
        c=IN IP4 143.163.151.157
        a=rtpmap:96 VP8/90000
        a=rtpmap:98 flexfec/90000
        a=fmtp:98; repair-window=200ms

7.2.  Example SDP for Flex FEC Protection with explicit signalling in
      the SDP

   In this example, we have one source video stream (ssrc:1234) and one
   FEC repair streams (ssrc:2345).  We form one FEC group with the
   "a=ssrc-group:FEC-FR 1234 2345" line.  The source and repair streams
   are multiplexed on different SSRCs.  The repair window is set to 200
   ms.

        v=0
        o=ali 1122334455 1122334466 IN IP4 fec.example.com
        s=2-D Parity FEC with no in band signalling Example
        t=0 0
        m=video 30000 RTP/AVP 100 110
        c=IN IP4 233.252.0.1/127
        a=rtpmap:100 MP2T/90000
        a=rtpmap:110 flexfec/90000
        a=fmtp:110 L:5; D:10; ToP:2; repair-window:200000
        a=ssrc:1234
        a=ssrc:2345
        a=ssrc-group:FEC-FR 1234 2345

   The format of the FEC header is shown in Figure 10.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |MSK|P|X|  CC   |M| PT recovery |        length recovery        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                          TS recovery                          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |   SSRC Count  |                    reserved                   |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                             SSRC_i                            |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |           SN base_i           |M or Mask[8-15]| N or Mask[0-7]|
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                    Mask [16-47] (optional)                    |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                                                               |
     +                    Mask [48-111] (optional)                   +
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                     ... next in SSRC_i ...                    |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

As defined within the packet, it's possible that FEC is protecting multiple SSRCs within the same flexible FEC packet. This potentially is a problem though. This may require a demuxer to fork a packet to multiple receivers and coordinate the sending of flexible fec packets upon multiple senders (which is a challenge). This it's recommended this be only used to encapsulate the packets for a single SSRC for SRST or used in the case of MRST streams for the layers that are fec'ed.

The codec name is flexfec. RTCRtpCodecParameters.parameters for flexfec needs a few definitions:
ulong long repair-window (sender/receiver param) - The time that spans the source packets and the corresponding repair packets. The size of the repair window is specified in microseconds.
ulong L - (optional sender/receiver param) - indicates the number of columns of the source block that are protected by this FEC block and it applies to all the source SSRCs. L is a positive integer.
ulong D - (optional sender/receiver param) - indicates the number of rows of the source block that are protected by this FEC block and it applies to all the source SSRCs. D is a positive integer.
ulong ToP - (optional sender/receiver param) - indicates the type of protection applied by the sender: 0 for 1-D interleaved FEC protection, 1 for 1-D non-interleaved FEC protection, and 2 for 2-D parity FEC protection. The ToP value of 3 is reserved for future uses.

{
"name": "flexfec",
...
"parameters": {
  "repair-window": 200000,
  "L": 5,
  "D": 10,
  "ToP": 2
  }
}