Add description of an API for controlling SDP codec negotiation

index.bs

@@ -111,7 +111,7 @@ At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the followin
     1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
     2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
     <!-- FIXME: Use pipeTo algorithm when available. -->
-    3. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal.
+    3. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`'s [=AbortController/signal=].
 
 <p class=note>
 Streams backpressure can optimize throughput while limiting processing and memory consumption by pausing data production as early as possible in a data pipeline.
@@ -130,6 +130,7 @@ The <dfn abstract-op>readEncodedData</dfn> algorithm is given a |rtcObject| as p
 1. Let |frame| be the newly produced frame.
 1. Set |frame|.`[[owner]]` to |rtcObject|.
 1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`.
+1. If |rtcObject|'s |transform|'s |options| contains {{RTCRtpScriptTransformParameters/acceptOnlyInputCodecs}} with the value True, and the {{RTCEncodedVideoFrameMetadata/mimeType}} of |frame| does not match any codec in |transform|'s |options|' |inputCodecs|, execute the [$writeEncodedData$] algorithm given |rtcObject| and |frame|.
 1. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`.
 
 The <dfn abstract-op>writeEncodedData</dfn> algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps:
@@ -146,7 +147,7 @@ The <dfn abstract-op>writeEncodedData</dfn> algorithm is given a |rtcObject| as
 
 On sender side, as part of [$readEncodedData$], frames produced by |rtcObject|'s encoder MUST be enqueued in |rtcObject|.`[[readable]]` in the encoder's output order.
 As [$writeEncodedData$] ensures that the transform cannot reorder frames, the encoder's output order is also the order followed by packetizers to generate RTP packets and assign RTP packet sequence numbers.
-The packetizer may expect the transformed data to still conform to the original format, e.g. a series of NAL units separated by Annex B start codes.
+The packetizer will expect the transformed data to conform to the format indicated by its {{RTCEncodedVideoFrameMetadata/mimeType}}.
 
 On receiver side, as part of [$readEncodedData$], frames produced by |rtcObject|'s packetizer MUST be enqueued in |rtcObject|.`[[readable]]` in the same encoder's output order.
 To ensure the order is respected, the depacketizer will typically use RTP packet sequence numbers to reorder RTP packets as needed before enqueuing frames in |rtcObject|.`[[readable]]`.
@@ -168,21 +169,21 @@ The `transform` setter steps are:
 4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`.
 5. Initialize |newPipeToController| to a new {{AbortController}}.
 6. If [=this=].`[[pipeToController]]` is not null, run the following steps:
-    1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`.signal.
+    1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=].
     2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`.
 7. Else, run the [$chain transform algorithm$] steps.
 8. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
 9. Set [=this=].`[[transform]]` to |transform|.
 10. Run the steps in the set of [$association steps$] of |transform| with [=this=].
 
 The <dfn abstract-op>chain transform algorithm</dfn> steps are defined as:
-1. If |newPipeToController|.signal is [=AbortSignal/aborted=], abort these steps.
+1. If |newPipeToController|'s [=AbortController/signal=] is [=AbortSignal/aborted=], abort these steps.
 2. [=ReadableStreamDefaultReader/Release=] |reader|.
 3. [=WritableStreamDefaultWriter/Release=] |writer|.
 4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`.
 <!-- FIXME: Use pipeTo algorithm when available. -->
-5. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal.
-6. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal.
+5. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|'s [=AbortController/signal=].
+6. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|'s [=AbortController/signal=].
 
 This algorithm is defined so that transforms can be updated dynamically.
 There is no guarantee on which frame will happen the switch from the previous transform to the new transform.
@@ -361,6 +362,25 @@ dictionary RTCEncodedVideoFrameMetadata {
 ### Members ### {#RTCEncodedVideoFrameMetadata-members}
 
 <dl dfn-for="RTCEncodedVideoFrameMetadata" class="dictionary-members">
+    <dt>
+        <dfn dict-member>frameId</dfn> <span class="idlMemberType">unsigned long long</span>
+    </dt>
+    <dd>
+        <p>
+            An identifier for the encoded frame, monotonically increasing in decode order. Its lower
+            16 bits match the frame_number of the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]], if present.
+            Only present for received frames if the Dependency Descriptor Header Extension is present.
+        </p>
+    </dd>
+    <dt>
+        <dfn dict-member>dependencies</dfn> <span class="idlMemberType">sequence&lt;unsigned long long&gt;</span>
+    </dt>
+    <dd>
+        <p>
+            List of frameIds of frames this frame references.
+            Only present for received frames if the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]] is present.
+        </p>
+    </dd>
     <dt>
         <dfn dict-member>synchronizationSource</dfn> <span class="idlMemberType">unsigned long</span>
     </dt>
@@ -772,15 +792,34 @@ interface KeyFrameRequestEvent : Event {
   constructor(DOMString type, optional DOMString rid);
   readonly attribute DOMString? rid;
 };
+
+dictionary RTCRtpScriptTransformParameters {
+  sequence&lt;RTCRtpCodec&gt; inputCodecs;
+  sequence&lt;RTCRtpCodec&gt; outputCodecs;
+  boolean acceptOnlyInputCodecs = false;
+};
+
+</pre>
+
+### Extension to RTCRtpCodec ### {#RTCRtpCodec-extensions}
+
+The member "packetizationMode" is added to the {{RTCRtpCodec}} definition; its value is a MIME type. When this is present, the rules for packetizing and depacketizing are those of the named MIME type. For codecs not supported by the platform, this MUST be present.
+
+<pre class='idl'>
+partial dictionary RTCRtpCodec {
+  DOMString packetizationMode;
+};
 </pre>
 
+
 ## Operations ## {#RTCRtpScriptTransform-operations}
 
 The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|)</code></dfn> constructor steps are:
 1. Set |t1| to an [=identity transform stream=].
 2. Set |t2| to an [=identity transform stream=].
 3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
 4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
+4. Set |this|.`[[options]]?` to the result of converting |options| to an RTCRtpScriptTransformParameters dictionary.
 5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
 6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
 7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
@@ -798,8 +837,17 @@ The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worke
 
 Each RTCRtpScriptTransform has the following set of [$association steps$], given |rtcObject|:
 1. Let |transform| be the {{RTCRtpScriptTransform}} object that owns the [$association steps$].
-1. Let |encoder| be |rtcObject|'s encoder if |rtcObject| is a {{RTCRtpSender}} or undefined otherwise.
-1. Let |depacketizer| be |rtcObject|'s depacketizer if |rtcObject| is a {{RTCRtpReceiver}} or undefined otherwise.
+1. If |rtcObject| is a {{RTCRtpSender}}, perform the following steps:
+    1. Let |encoder| be |rtcObject|'s encoder.
+    1. If `[[options]]` contains |outputCodecs|, then for each element |codec| in |outputCodecs|:
+        1. If |codec| is not in |rtcObject|'s {{RTCRtpSender/[[SendCodecs]]}}, add it.
+        1. Set the "enabled" flag of |codec| in |rtcObject|'s {{RTCRtpSender/[[SendCodecs]]}} to True.
+
+1. If |rtcObject| is a {{RTCRtpReceiver}}, perform the following steps:
+    1. Let |depacketizer| be |rtcObject|'s depacketizer
+    1. If `[[options]]` contains |inputCodecs|, then for each element |codec| in |inputCodecs|:
+        1. If |codec| is not in |rtcObject|'s {{RTCRtpReceiver/[[ReceiveCodecs]]}}, add it.
+        1. Set the "enabled" flag of |codec| in |rtcObject|'s {{RTCRtpReceiver/[[ReceiveCodecs]]}} to True.
 1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
     1. Let |transformer| be the {{RTCRtpScriptTransformer}} object associated to |transform|.
     1. Set |transformer|.`[[encoder]]` to |encoder|.

sdp-explainer.md

@@ -0,0 +1,140 @@
+# SDP negotiation in Encoded Transform
+
+The original definition of encoded transform did not consider negotiation; the frames on the "transformed" side went out stamped with the payload type of the frame that came in on the "non-transformed" side (and vice versa for the receiver).
+
+This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire may not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging SFrame encrypted media between two endpoints from different application providers, or when exchanging SFrame encrypted content via an SFU that expects to be able to decode or modify the media.
+
+(The latter is exactly what SFrame is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.)
+
+This problem is even more acute if an interface resembling RTCRtpScriptTransform is used to add support for codecs not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to associate the payload type of incoming packets with the right decoder.
+
+For example, it's been proposed to add [Lyra](https://github.com/google/lyra) to WebRTC using an implementation in WASM; a working example using SDP munging can be found on the
+[Meetecho blog](https://www.meetecho.com/blog/playing-with-lyra/).
+
+However, this API proposal does not directly address that use case at the moment.
+
+# Requirements for an SDP negotiation API
+The following things need to be available on such an API:
+1. Before SDP negotiation, the application must be able to specify one or more new media types that one wants to negotiate for. As a point of illustration, this document uses the type "video/new-codec".
+2. After SDP negotiation, the application must be able to identify if negotiation has succeeded or failed, and what payload type has been assigned for the new media type.
+3. Before sending frames, the application must be able to inform the RTP sender of what kind of packetization to use on the outgoing frames.
+4. Before receiving frames, the application must be able to inform the RTP receiver of what kind of depacketization to use on the incoming frames.
+5. When transforming frames, the sending application must be able to mark the transformed frames with the negotiated payload type before sending.
+6. When transforming frames, the receiving application must be able to check that the incoming frame has the negotiated payload type, and (if reenqueueing the frame after transformation) mark the transformed frame with the appropriate payload type for decoding within the RTPReceiver.
+
+# API description
+
+## Codec description
+For codec description, we reuse the dictionary RTCRtpCodecCapability, but add a new field describing the
+packetization mode to be used.
+
+The requirements on the parameters are:
+- either mimetype or fmtp parameters must be different from any existing capability
+- the packetization mode must identify a mode known to the UA.
+
+When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getParameters().
+
+## Describing the input and output codecs of transforms
+
+We extend the RTCRtpCodecCapability object with a "packetizer" element, which identifies a media type with a packetizer
+known to the platform.
+
+We extend the RTCRTPScriptTransform object's constructor with a fourth argument of type CodecInformation, with the following IDL definition:
+
+```
+dictionary CodecInformation {
+  sequence<RTCRtpCodecCapabilityWithPacketization> inputCodecs;
+  sequence<RTCRtpCodecCapabilityWithPacketization> outputCodecs;
+  bool acceptOnlyInputCodecs = false;
+}
+```
+The inputCodecs member describe the media types the transform is prepared to process. Any frame of a format
+not listed will be passed to the output of the transform without modification.
+
+The outputCodecs describes the media types the transform may produce.
+
+NOTE: The inputCodecs has two purposes in the "Transform proposal" below - it gives codecs to
+negotiate in the SDP, and it serves to filter the frame types that the transform will process.
+
+In order to be able to use the filtering function, the "acceptOnlyInputCodecs" has to be set to true;
+if it is false, all frames are delivered to the transform.
+
+## For SDP negotiation
+
+When the PeerConnection generates an offer or an answer:
+
+* If a transform is set on the sender, the process for generating an offer or answer will add the codecs 
+  listed in the transform's outputCodecs to the list of codecs available for sending.
+
+* If a transform is set on the receiver, the process for generating an offer or answer will add the codecs 
+  listed in the transform's inputCodecs to the list of codecs available for receiving.
+
+When the transform attribute of a sender or receiver is changed, and the relevant codec list changes, the "negotiationneeded" event fires.
+
+## Existing APIs that will be used together with the new APIs
+- Basic establishing of EncodedTransform
+- getParameters() to get results of codec negotiation
+- encoded frame SetMetadata, to set the payload type for processed frames
+- setCodecPreferences, to say which codecs (old or new) are preferred for reception
+
+
+# Example code
+
+```js
+const worker = new Worker(`data:text/javascript,(${work.toString()})()`);
+
+// At sender side.
+const sender = pc.addTrack(track);
+const {codecs} = RTCRtpSender.getCapabilities();
+const vp8 = codecs.find(({mimeType}) => mimeType == "video/vp8");
+sender.transform = new RTCRtpScriptTransform(worker, {
+  inputCodecs: [vp8],
+  outputCodecs: [{mimeType: “video/x-encrypted”,
+                  packetizationMode: "video/sframe"}]
+});
+
+// At receiver side.
+pc.ontrack = ({receiver}) => {
+  const {codecs} = receiver.getParameters();
+  const customCodec = codecs.find(({mimeType}) => mimeType == "video/x-encrypted");
+  if (customCodec) {
+    receiver.transform = new RTCRtpScriptTransform(worker, {
+      inputCodecs: [customCodec],
+      outputCodecs: [{mimeType: "video/vp8"}]
+    });
+  }
+}
+
+// Same worker can handle both sides.
+function work() {
+  onrtctransform = async ({transformer: {readable, writable}, {inputCodecs, outputCodecs}}) => {
+    const [outputCodec] = outputCodecs;
+    await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable);
+    function transform(frame, controller) {
+      // transform chunk
+      let metadata = frame.metadata();
+      const inputCodec = inputCodecs.find((mimeType) => mimeType == metadata.mediaType);
+
+      if (inputCodec && outputCodec.mimeType == "video/x-encrypted") {
+        encryptBody(frame, inputCodec);
+        metadata.mediaType = outputCodec.mimeType;
+        frame.setMetadata(metadata);
+      } else if (inputCodec.mimeType == "video/x-encrypted") {
+        decryptBody(frame, outputCodec);
+        metadata.mediaType = outputCodec.mimeType;
+        frame.setMetadata(metadata);
+      }
+      controller.enqueue(frame);
+    }
+  }
+}
+```
+# Frequently asked questions
+
+1.  Q: My application wants to send frames with multiple packetizers. How do I accomplish that?
+
+    A: Use multiple media types. Each will be assigned a payload type. Mark each frame with the media type they need to be packetized as.
+
+1.  Q: What is the relationship between this proposal and the IETF standard for SFrame?
+
+    A: This proposal is intended to make it possible to implement the IETF standard for SFrame in Javascript, with only the packetizer/depacketizer being updated to support the SFrame packetization rules. It is also intended to make it possible to perform other forms of transform, including the ones that are presently deployed in the field, while marking the SDP with a truthful statement of its content.