From a14a85528a636af44d3c8a6dd2b4103465e10138 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 30 May 2023 15:39:54 +0200 Subject: [PATCH 01/42] Create sdp-explainer.md --- sdp-explainer.md | 119 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 sdp-explainer.md diff --git a/sdp-explainer.md b/sdp-explainer.md new file mode 100644 index 0000000..fc43f96 --- /dev/null +++ b/sdp-explainer.md @@ -0,0 +1,119 @@ +# SDP negotiation in Encoded Transform + +The original definition of encoded transform did not consider negotation; the frames on the "transformed" side went out stamped with the payload type of the frame that came in on the "non-transformed" side (and vice versa for the receiver). + +This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire do not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging Sframe encrypted media between two endpoints from different application providers, or when exchanging Sframe encrypted content via an SFU that expects to be able to decode or modify the media. + +(The latter is exactly what Sframe is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.) + +This problem is even more acute when the Encoded Transform is used to add support for payload types not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to demultiplex the incoming packets correctly and route them to the right decoder. + +# Requirements for an SDP negotiation API +The following things need to be available on such an API: +1. Before SDP negotiation, the application must be able to specify one or more new media types that one wants to negotiate for. As a point of illustration, this document uses the type "video/new-codec". +2. After SDP negotiation, the application must be able to identify if negotiation has succeeded or failed, and what payload type has been assigned for the new media type. +3. Before sending frames, the application must be able to inform the RTP sender of what kind of packetization to use on the outgoing frames. +4. Before receiving frames, the application must be able to inform the RTP receiver of what kind of depacketization to use on the incoming frames. +5. When transforming frames, the sending application must be able to mark the transformed frames with the negotiated payload type before sending. +6. When transforming frames, the receiving application must be able to check that the incoming frame has the negotiated payload type, and (if reenqueueing the frame after transformation) mark the transformed frame with the appropriate payload type for decoding within the RTPReceiver. + +# API description + +## Codec description +For codec description, we reuse the dictionary RTCRtpCodecCapability, but add a DOMString that identifies the packetization mode. + +The requirements on the parameters are: +- either mimetype or fmtp parameters must be different from any existing capability +- if the mimetype is not a known codec, the packetizationMode member MUST be given, and be the mimetype of a known codec. + +When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getCapabilities(). + + +## For SDP negotiation +``` +PeerConneciton.AddSendCodecCapability(DOMString kind, CodecCapability capability) +PeerConnection.AddReceiveCodecCapability(DOMString kind, CodecCapability capability) +``` +These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to GetParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) + +NOTE: The codecs will not show up on the global GetCapaiblity functions, since these functions can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. + +## For sending +The RTCRtpSender’s encoder (if present) will be configured to use a specific codec from CodecCapabilities by a new call: +``` +RTCRtpSender.SetEncodingCodec(RTCCodecParameters parameters) +``` +This sets the MIME type of the codec to encode to, and the payload type that will be put on frames produced by that encoder. This codec must be one supported by the platform (not the “novel” codec), and the PT does not need to be one negotiated in the SDP offer/answer. + +When configuring the transform post negotiation, the app MUST retrieve the PTs negotiated for the connection, and identify the PT for the custom codec. + +When transforming frames, the transformer configured MUST, in addition to modifying the payload, modify the metadata to have the negotiated PT for the custom codec. + +The packetizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. (This assumes that packetization is independent of FMTP) +## For receiving +The depacketizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. + +The decoder can be configured to accept a given PT as indicating a given codec format by the new API call: +``` +AddDecodingCodec(CodecParameters parameters) +``` +This does not alter the handling of any otherwise-configured PT, but adds a handler for this specific PT. + +On seeing a custom codec in the PT for an incoming frame, if the frame is to be delivered to the corresponding decoder, the encoded frame transform MUST, in addition to transforming the payload, set the PT for the frame to a PT that is understood by the decoder - either by being negotiated or by having been added by AddDecodingCodec. + +## Existing APIs that will be used together with the new APIs +- Basic establishing of EncodedTransform +- + +# Example code +(This is incomplete) +``` +customCodec = { + mimeType: “application/x-encrypted”, + clockRate: 90000, + fmtp = “encapsulated-codec=vp8”, + packetizationMode = “video/vp8”, +}; + +// At sender side +RTCRtpSender.AddCodecCapability(customCodec); +sender = pc.AddTrack(videotrack); +// Negotiate as usual +for (codec in sender.getParameters(‘video’).codecs) { + if (codec.mimeType == “application/x-encrypted”) { + encryptedPT = codec.payloadType; + } +} +(readable, writable) = sender.getEncodedStreams(); + +readable.pipeThrough(new TransformStream( + transform: (frame) => { + encryptBody(frame); + metadata = frame.metadata(); + metadata.pt = encryptedPT; + frame.setMetadata(metadata); + writable.write(frame); + } +}).pipeTo(writable); + +// At receiver side +RTCRtpReceiver.AddCodecCapability(customCodec); +pc.ontrack = (receiver) => { + + for (codec in receiver.getParameters.codecs) { + if (codec.mimeType == “application/x-encrypted”) { + encryptedPT = codec.payloadType; + } + } + receiver.addDecodeCapability({mimeType: video/vp8, payloadType=208}); + (readable, writable) = receiver.getEncodedStreams(); + readable.pipeThrough(new TransformStream( + transform: (frame) => { + decryptBody(frame); + metadata = frame.metadata(); + metadata.payloadType = 208; + writable.write(frame); + } + }).pipeTo(writable); +}; +``` From 005121cd7f52134d84165cd332ccc1e5a5a12ec6 Mon Sep 17 00:00:00 2001 From: Florent Castelli Date: Tue, 30 May 2023 10:16:46 -0700 Subject: [PATCH 02/42] Fix typos --- sdp-explainer.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index fc43f96..74cc25d 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -31,12 +31,12 @@ When a codec capability is added, the SDP machinery will negotiate these codecs ## For SDP negotiation ``` -PeerConneciton.AddSendCodecCapability(DOMString kind, CodecCapability capability) +PeerConnection.AddSendCodecCapability(DOMString kind, CodecCapability capability) PeerConnection.AddReceiveCodecCapability(DOMString kind, CodecCapability capability) ``` These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to GetParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) -NOTE: The codecs will not show up on the global GetCapaiblity functions, since these functions can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. +NOTE: The codecs will not show up on the global GetCapability functions, since these functions can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. ## For sending The RTCRtpSender’s encoder (if present) will be configured to use a specific codec from CodecCapabilities by a new call: @@ -79,7 +79,7 @@ customCodec = { RTCRtpSender.AddCodecCapability(customCodec); sender = pc.AddTrack(videotrack); // Negotiate as usual -for (codec in sender.getParameters(‘video’).codecs) { +for (codec in sender.getParameters().codecs) { if (codec.mimeType == “application/x-encrypted”) { encryptedPT = codec.payloadType; } @@ -100,18 +100,18 @@ readable.pipeThrough(new TransformStream( RTCRtpReceiver.AddCodecCapability(customCodec); pc.ontrack = (receiver) => { - for (codec in receiver.getParameters.codecs) { + for (codec in receiver.getParameters().codecs) { if (codec.mimeType == “application/x-encrypted”) { encryptedPT = codec.payloadType; } } - receiver.addDecodeCapability({mimeType: video/vp8, payloadType=208}); + receiver.addDecodeCapability({mimeType: video/vp8, payloadType=encryptedPT}); (readable, writable) = receiver.getEncodedStreams(); readable.pipeThrough(new TransformStream( transform: (frame) => { decryptBody(frame); metadata = frame.metadata(); - metadata.payloadType = 208; + metadata.payloadType = encryptedPT; writable.write(frame); } }).pipeTo(writable); From 8c5ec201fcc4b6ff98301dda727954ffb87c29a5 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 2 Jun 2023 13:22:27 +0200 Subject: [PATCH 03/42] Add decryptedPT, erroneous frame handling --- sdp-explainer.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 74cc25d..e667e55 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -63,7 +63,7 @@ On seeing a custom codec in the PT for an incoming frame, if the frame is to be ## Existing APIs that will be used together with the new APIs - Basic establishing of EncodedTransform -- +- getParameters() to get results of codec negotiation # Example code (This is incomplete) @@ -96,7 +96,8 @@ readable.pipeThrough(new TransformStream( } }).pipeTo(writable); -// At receiver side +// At receiver side. +const decryptedPT = 208; // Can be negotiated PT or locally-valid RTCRtpReceiver.AddCodecCapability(customCodec); pc.ontrack = (receiver) => { @@ -105,13 +106,16 @@ pc.ontrack = (receiver) => { encryptedPT = codec.payloadType; } } - receiver.addDecodeCapability({mimeType: video/vp8, payloadType=encryptedPT}); + receiver.addDecodeCapability({mimeType: video/vp8, payloadType=decryptedPT}); (readable, writable) = receiver.getEncodedStreams(); readable.pipeThrough(new TransformStream( transform: (frame) => { + if (frame.metadata().payloadType != encryptedPT) { + continue; // Ignore all frames for the wrong payload. + } decryptBody(frame); metadata = frame.metadata(); - metadata.payloadType = encryptedPT; + metadata.payloadType = decryptedPT; writable.write(frame); } }).pipeTo(writable); From 9c7ba95899f6991c762f9d5937f667e81f0a7074 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 13 Jun 2023 14:35:53 -0400 Subject: [PATCH 04/42] Add specification --- index.bs | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 67 insertions(+), 1 deletion(-) diff --git a/index.bs b/index.bs index 8689e85..e525d94 100644 --- a/index.bs +++ b/index.bs @@ -61,21 +61,87 @@ This specification shows the IDL extensions for WebRTC. It uses an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to insert the processing into the pipeline. +## RTCPeerConnection extensions ## {#rtcpeerconnection-extensions} +In order to correctly identify the transformed content to the other end +of the connection, new interfaces are added on {{RTCPeerConnection}} to +allow the declaration and negotiation of new media types. + +
+// New methods for RTCPeerConnection
+partial interface RTCPeerConnection {
+   undefined AddSendCodecCapability(DOMString kind,
+                                    RTCRtpCodecCapability capability);
+   undefined AddReceiveCodecCapability(DOMString kind,
+                                    RTCRtpCodecCapability capability);
+};
+
+ +In addition, add two new internal slots to the RTCPeerConnection, called +"CustomSendCodecs" and "CustomReceiveCodecs", initially empty. + +### Methods ### {#rtcpeerconnection-methods} + +#### AddSendCodecCapability #### {#add-send-codec-capability} + +1. Let 'capability' be the "capability" argument +1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. +1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. +1. Add 'capability' to the PeerConnections's [[ CustomSendCodecs]] internal slot. + +#### AddReceiveCodecCapability #### {#add-receive-codec-capability} + +1. Let 'capability' be the "capability" argument +1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. +1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. +1. Add 'capability' to the PeerConnections's [[ CustomReceiveCodecs]] internal slot. + +## RTCRtpCodecCapability extensions ## {#rtc-rtp-codec-capability-extensions} + +
+partial dictionary RTCRtpCodecCapability {
+  DOMString packetizationMode;
+};
+
+ +### Attributes ### {#rtc-rtp-codec-capability-attributes} +#### packetizationMode #### {#packetization-mode} +Defines the packetization mode to use for the frame being packetized or depacketized. MUST be the name of a codec known to the browser implementation. + + +## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions} 
+
 typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
 
-// New methods for RTCRtpSender and RTCRtpReceiver
+// New methods and attributes for RTCRtpSender and RTCRtpReceiver
 partial interface RTCRtpSender {
     attribute RTCRtpTransform? transform;
+    undefined SetEncodingCodec(RTCRtpCodecParameters parameters);
 };
 
 partial interface RTCRtpReceiver {
     attribute RTCRtpTransform? transform;
+    undefined AddDecodingCodec(RTCRtpCodecParameters parameters);
 };
## Extension operation ## {#operation} +### Media type negotiation ### {#sdp-negotiation} +The following modifications are applied to the session negotation procedure: + +1. The CustomSendCodecs is included in the list of codecs which the user agent + is currently prepared to send; the CustomReceiveCodecs is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. + +1. In the final steps to create an offer, step 2.2 is changed to filter the list of codecs against + the union of the list of implemented send codecs and the [[ CustomSendCodecs]] slot, and step 2.3 is changed to filter against the union of the list of implemented receive codecs and the [[ CustomReceiveCodecs]] slot. + +1. In the final steps to create an answer, the same modification is done to the filtering in steps 1.2 and 1.3. + + + + +### Codec initialization ### {#codec-initialization} At the time when a codec is initialized as part of the encoder, and the corresponding flag is set in the {{RTCPeerConnection}}'s {{RTCConfiguration}} argument, ensure that the codec is disabled and produces no output. From 6c6ca270856af4e4134352f4a313c96b54d4f7ab Mon Sep 17 00:00:00 2001 From: Elad Alon Date: Wed, 14 Jun 2023 12:18:18 +0200 Subject: [PATCH 05/42] Fix internal slots (or die trying) --- index.bs | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/index.bs b/index.bs index e525d94..61a7ebc 100644 --- a/index.bs +++ b/index.bs @@ -77,7 +77,8 @@ partial interface RTCPeerConnection { In addition, add two new internal slots to the RTCPeerConnection, called -"CustomSendCodecs" and "CustomReceiveCodecs", initially empty. +`[[CustomSendCodecs]]` and `[[CustomReceiveCodecs]]`, +initially empty. ### Methods ### {#rtcpeerconnection-methods} @@ -86,14 +87,14 @@ In addition, add two new internal slots to the RTCPeerConnection, called 1. Let 'capability' be the "capability" argument 1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. 1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. -1. Add 'capability' to the PeerConnections's [[ CustomSendCodecs]] internal slot. +1. Add 'capability' to the PeerConnections's {{[[CustomSendCodecs]]}} internal slot. #### AddReceiveCodecCapability #### {#add-receive-codec-capability} 1. Let 'capability' be the "capability" argument 1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. 1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. -1. Add 'capability' to the PeerConnections's [[ CustomReceiveCodecs]] internal slot. +1. Add 'capability' to the PeerConnections's {{[[CustomReceiveCodecs]]}} internal slot. ## RTCRtpCodecCapability extensions ## {#rtc-rtp-codec-capability-extensions} @@ -130,11 +131,11 @@ partial interface RTCRtpReceiver { ### Media type negotiation ### {#sdp-negotiation} The following modifications are applied to the session negotation procedure: -1. The CustomSendCodecs is included in the list of codecs which the user agent - is currently prepared to send; the CustomReceiveCodecs is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. +1. The {{[[CustomSendCodecs]]}} is included in the list of codecs which the user agent + is currently prepared to send; the {{[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. 1. In the final steps to create an offer, step 2.2 is changed to filter the list of codecs against - the union of the list of implemented send codecs and the [[ CustomSendCodecs]] slot, and step 2.3 is changed to filter against the union of the list of implemented receive codecs and the [[ CustomReceiveCodecs]] slot. + the union of the list of implemented send codecs and the {{[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the list of implemented receive codecs and the {{[[CustomReceiveCodecs]]}} slot. 1. In the final steps to create an answer, the same modification is done to the filtering in steps 1.2 and 1.3. From d3c6ec588944997131c63875d34ec9c0db52294d Mon Sep 17 00:00:00 2001 From: Elad Alon Date: Wed, 14 Jun 2023 16:15:57 +0200 Subject: [PATCH 06/42] Fix linker --- index.bs | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/index.bs b/index.bs index 61a7ebc..e32de50 100644 --- a/index.bs +++ b/index.bs @@ -77,7 +77,8 @@ partial interface RTCPeerConnection { In addition, add two new internal slots to the RTCPeerConnection, called -`[[CustomSendCodecs]]` and `[[CustomReceiveCodecs]]`, +{{RTCPeerConnection/[[CustomSendCodecs]]}} +and {{RTCPeerConnection/[[CustomReceiveCodecs]]}}, initially empty. ### Methods ### {#rtcpeerconnection-methods} @@ -87,14 +88,14 @@ initially empty. 1. Let 'capability' be the "capability" argument 1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. 1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. -1. Add 'capability' to the PeerConnections's {{[[CustomSendCodecs]]}} internal slot. +1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot. #### AddReceiveCodecCapability #### {#add-receive-codec-capability} 1. Let 'capability' be the "capability" argument 1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. 1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. -1. Add 'capability' to the PeerConnections's {{[[CustomReceiveCodecs]]}} internal slot. +1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot. ## RTCRtpCodecCapability extensions ## {#rtc-rtp-codec-capability-extensions} @@ -131,11 +132,11 @@ partial interface RTCRtpReceiver { ### Media type negotiation ### {#sdp-negotiation} The following modifications are applied to the session negotation procedure: -1. The {{[[CustomSendCodecs]]}} is included in the list of codecs which the user agent - is currently prepared to send; the {{[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. +1. The {{RTCPeerConnection/[[CustomSendCodecs]]}} is included in the list of codecs which the user agent + is currently prepared to send; the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. 1. In the final steps to create an offer, step 2.2 is changed to filter the list of codecs against - the union of the list of implemented send codecs and the {{[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the list of implemented receive codecs and the {{[[CustomReceiveCodecs]]}} slot. + the union of the list of implemented send codecs and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the list of implemented receive codecs and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot. 1. In the final steps to create an answer, the same modification is done to the filtering in steps 1.2 and 1.3. From 1342689cbde131e00d7f8bd1eda25dcb1fb1cb50 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Wed, 14 Jun 2023 13:19:51 -0400 Subject: [PATCH 07/42] More sensible errors to throw --- index.bs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/index.bs b/index.bs index e32de50..b5bc29f 100644 --- a/index.bs +++ b/index.bs @@ -86,15 +86,15 @@ initially empty. #### AddSendCodecCapability #### {#add-send-codec-capability} 1. Let 'capability' be the "capability" argument -1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. -1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. +1. If 'capability' is a known codec, throw an IllegalModification error. +1. If 'packetizationMode' is defined, and is not a known codec, throw a SyntaxError DOMException. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot. #### AddReceiveCodecCapability #### {#add-receive-codec-capability} 1. Let 'capability' be the "capability" argument -1. If 'capability' is a known codec, throw an AlreadyDefined DOMError. -1. If 'packetizationMode' is defined, and is not a known codec, throw an IllegalArgument DOMError. +1. If 'capability' is a known codec, throw an IllegalModification error. +1. If 'packetizationMode' is defined, and is not a known codec, throw a SyntaxError DOMException. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot. ## RTCRtpCodecCapability extensions ## {#rtc-rtp-codec-capability-extensions} @@ -109,7 +109,6 @@ partial dictionary RTCRtpCodecCapability { #### packetizationMode #### {#packetization-mode} Defines the packetization mode to use for the frame being packetized or depacketized. MUST be the name of a codec known to the browser implementation. - ## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions} 
 

From bf6e469d172774f8374f89c9bbe2da98293523c4 Mon Sep 17 00:00:00 2001
From: Harald Alvestrand 
Date: Thu, 15 Jun 2023 11:26:10 -0400
Subject: [PATCH 08/42] Lowercase functions, add linkages

---
 index.bs         | 24 ++++++++++++++++++------
 sdp-explainer.md | 22 +++++++++++++---------
 2 files changed, 31 insertions(+), 15 deletions(-)

diff --git a/index.bs b/index.bs
index b5bc29f..98a1e78 100644
--- a/index.bs
+++ b/index.bs
@@ -117,27 +117,39 @@ typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
 // New methods and attributes for RTCRtpSender and RTCRtpReceiver
 partial interface RTCRtpSender {
     attribute RTCRtpTransform? transform;
-    undefined SetEncodingCodec(RTCRtpCodecParameters parameters);
+    undefined setEncodingCodec(RTCRtpCodecParameters parameters);
 };
 
 partial interface RTCRtpReceiver {
     attribute RTCRtpTransform? transform;
-    undefined AddDecodingCodec(RTCRtpCodecParameters parameters);
+    undefined addDecodingCodec(RTCRtpCodecParameters parameters);
 };
+### New methods ### {#sender-receiver-new-methods} +
+
setEncodingCodec
+
+When called, switch the encoder in the RTCRtpSender to produce frames in the format indicated by the `parameters` argument. +
+
addDecodingCodec
+
+When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument, and initialize the demultiplexer part of the RTCRtpReceiver to forward frames with the payload type identical to `parameters.payloadType` to this decoder. +
+
+ ## Extension operation ## {#operation} ### Media type negotiation ### {#sdp-negotiation} The following modifications are applied to the session negotation procedure: 1. The {{RTCPeerConnection/[[CustomSendCodecs]]}} is included in the list of codecs which the user agent - is currently prepared to send; the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. + is currently capable of sending; the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. -1. In the final steps to create an offer, step 2.2 is changed to filter the list of codecs against - the union of the list of implemented send codecs and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the list of implemented receive codecs and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot. +1. In the [=final steps to create an offer=], step 2.2 is changed to filter the list of codecs against + the union of the [=RTCRtpSender/list of implemented send codecs=] and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the [=RTCRtpReceiver/list of implemented receive codecs=] and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot. -1. In the final steps to create an answer, the same modification is done to the filtering in steps 1.2 and 1.3. +1. In the [=final steps to create an answer=], the same modification is done to the filtering in steps 1.2 and 1.3. diff --git a/sdp-explainer.md b/sdp-explainer.md index e667e55..6aecb4d 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -31,8 +31,8 @@ When a codec capability is added, the SDP machinery will negotiate these codecs ## For SDP negotiation ``` -PeerConnection.AddSendCodecCapability(DOMString kind, CodecCapability capability) -PeerConnection.AddReceiveCodecCapability(DOMString kind, CodecCapability capability) +pc.addSendCodecCapability(DOMString kind, CodecCapability capability) +pc.addReceiveCodecCapability(DOMString kind, CodecCapability capability) ``` These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to GetParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) @@ -41,7 +41,7 @@ NOTE: The codecs will not show up on the global GetCapability functions, since t ## For sending The RTCRtpSender’s encoder (if present) will be configured to use a specific codec from CodecCapabilities by a new call: ``` -RTCRtpSender.SetEncodingCodec(RTCCodecParameters parameters) +sender.setEncodingCodec(RTCCodecParameters parameters) ``` This sets the MIME type of the codec to encode to, and the payload type that will be put on frames produced by that encoder. This codec must be one supported by the platform (not the “novel” codec), and the PT does not need to be one negotiated in the SDP offer/answer. @@ -50,12 +50,13 @@ When configuring the transform post negotiation, the app MUST retrieve the PTs n When transforming frames, the transformer configured MUST, in addition to modifying the payload, modify the metadata to have the negotiated PT for the custom codec. The packetizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. (This assumes that packetization is independent of FMTP) + ## For receiving The depacketizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. The decoder can be configured to accept a given PT as indicating a given codec format by the new API call: ``` -AddDecodingCodec(CodecParameters parameters) +receiver.addDecodingCodec(CodecParameters parameters) ``` This does not alter the handling of any otherwise-configured PT, but adds a handler for this specific PT. @@ -69,21 +70,24 @@ On seeing a custom codec in the PT for an incoming frame, if the frame is to be (This is incomplete) ``` customCodec = { - mimeType: “application/x-encrypted”, + mimeType: “video/x-encrypted”, clockRate: 90000, fmtp = “encapsulated-codec=vp8”, packetizationMode = “video/vp8”, }; // At sender side -RTCRtpSender.AddCodecCapability(customCodec); +pc.addSendCodecCapability('video', customCodec); sender = pc.AddTrack(videotrack); // Negotiate as usual for (codec in sender.getParameters().codecs) { - if (codec.mimeType == “application/x-encrypted”) { + if (codec.mimeType == “video/x-encrypted”) { encryptedPT = codec.payloadType; + } else if (codec.mimeType == 'video/vp8') { + encodingCodec = codec; } } +sender.setEncodingCodec(encodingCodec); (readable, writable) = sender.getEncodedStreams(); readable.pipeThrough(new TransformStream( @@ -98,7 +102,7 @@ readable.pipeThrough(new TransformStream( // At receiver side. const decryptedPT = 208; // Can be negotiated PT or locally-valid -RTCRtpReceiver.AddCodecCapability(customCodec); +pc.addReceiveCodecCapability('video', customCodec); pc.ontrack = (receiver) => { for (codec in receiver.getParameters().codecs) { @@ -106,7 +110,7 @@ pc.ontrack = (receiver) => { encryptedPT = codec.payloadType; } } - receiver.addDecodeCapability({mimeType: video/vp8, payloadType=decryptedPT}); + receiver.addDecodingCodec({mimeType: 'video/vp8', payloadType: decryptedPT}); (readable, writable) = receiver.getEncodedStreams(); readable.pipeThrough(new TransformStream( transform: (frame) => { From 9d8212cc386cd2f069fed34a914c217cc8fbb60f Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 20 Jun 2023 10:54:54 +0100 Subject: [PATCH 09/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 6aecb4d..61ca212 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -34,7 +34,7 @@ When a codec capability is added, the SDP machinery will negotiate these codecs pc.addSendCodecCapability(DOMString kind, CodecCapability capability) pc.addReceiveCodecCapability(DOMString kind, CodecCapability capability) ``` -These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to GetParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) +These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to getParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) NOTE: The codecs will not show up on the global GetCapability functions, since these functions can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. From cba1936de54ce0f967da1c687022674f37b700d2 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 20 Jun 2023 10:55:14 +0100 Subject: [PATCH 10/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 61ca212..992604c 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -36,7 +36,7 @@ pc.addReceiveCodecCapability(DOMString kind, CodecCapability capability) ``` These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to getParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) -NOTE: The codecs will not show up on the global GetCapability functions, since these functions can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. +NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. ## For sending The RTCRtpSender’s encoder (if present) will be configured to use a specific codec from CodecCapabilities by a new call: From 871a6bb2331f408eef16e7b444c7ce02d314c4e4 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 20 Jun 2023 10:55:30 +0100 Subject: [PATCH 11/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 992604c..583410b 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -47,7 +47,7 @@ This sets the MIME type of the codec to encode to, and the payload type that wil When configuring the transform post negotiation, the app MUST retrieve the PTs negotiated for the connection, and identify the PT for the custom codec. -When transforming frames, the transformer configured MUST, in addition to modifying the payload, modify the metadata to have the negotiated PT for the custom codec. +When transforming frames, the transformer configured MUST, in addition to modifying the payload, modify the metadata to have the negotiated payload type for the custom codec. The packetizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. (This assumes that packetization is independent of FMTP) From 2cd3e3960d0f08f886f4362eac2be5ed2109eff2 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 20 Jun 2023 10:56:10 +0100 Subject: [PATCH 12/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 1 - 1 file changed, 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 583410b..d389c35 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -104,7 +104,6 @@ readable.pipeThrough(new TransformStream( const decryptedPT = 208; // Can be negotiated PT or locally-valid pc.addReceiveCodecCapability('video', customCodec); pc.ontrack = (receiver) => { - for (codec in receiver.getParameters().codecs) { if (codec.mimeType == “application/x-encrypted”) { encryptedPT = codec.payloadType; From ebdb6fc6d7d39f3f038e604fdf1093be45ee6efc Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 29 Aug 2023 14:13:56 +0200 Subject: [PATCH 13/42] Some review comments addressed on index.bs --- index.bs | 29 +++++++++++++++++++---------- 1 file changed, 19 insertions(+), 10 deletions(-) diff --git a/index.bs b/index.bs index 98a1e78..db7f898 100644 --- a/index.bs +++ b/index.bs @@ -69,10 +69,8 @@ allow the declaration and negotiation of new media types. 
 // New methods for RTCPeerConnection
 partial interface RTCPeerConnection {
-   undefined AddSendCodecCapability(DOMString kind,
-                                    RTCRtpCodecCapability capability);
-   undefined AddReceiveCodecCapability(DOMString kind,
-                                    RTCRtpCodecCapability capability);
+   undefined AddSendCodecCapability(RTCRtpCodecCapability capability);
+   undefined AddReceiveCodecCapability(RTCRtpCodecCapability capability);
 };
@@ -86,14 +84,16 @@ initially empty. #### AddSendCodecCapability #### {#add-send-codec-capability} 1. Let 'capability' be the "capability" argument -1. If 'capability' is a known codec, throw an IllegalModification error. +1. If 'capability' is identical to the capability describing a previously configured codec, + throw an IllegalModification error. 1. If 'packetizationMode' is defined, and is not a known codec, throw a SyntaxError DOMException. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot. #### AddReceiveCodecCapability #### {#add-receive-codec-capability} 1. Let 'capability' be the "capability" argument -1. If 'capability' is a known codec, throw an IllegalModification error. +1. If 'capability' is identical to the capability describing a previously configured codec, + throw an IllegalModification error. 1. If 'packetizationMode' is defined, and is not a known codec, throw a SyntaxError DOMException. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot. @@ -107,7 +107,9 @@ partial dictionary RTCRtpCodecCapability { ### Attributes ### {#rtc-rtp-codec-capability-attributes} #### packetizationMode #### {#packetization-mode} -Defines the packetization mode to use for the frame being packetized or depacketized. MUST be the name of a codec known to the browser implementation. +Defines the packetization mode to use for the frame being packetized or depacketized. MUST be the mimeType from +the capability describing a codec in the [=list of implemented send codecs=] (for packetizing) or the +[=list of implemented receive codecs=] (for depacketizing). ## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions} 
@@ -122,8 +124,7 @@ partial interface RTCRtpSender {
 
 partial interface RTCRtpReceiver {
     attribute RTCRtpTransform? transform;
-    undefined addDecodingCodec(RTCRtpCodecParameters parameters);
-};
+    undefined addDecodingCodec(RTCRtpCodecParameters parameters);};
### New methods ### {#sender-receiver-new-methods} @@ -134,10 +135,18 @@ When called, switch the encoder in the RTCRtpSender to produce frames in the for
addDecodingCodec
-When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument, and initialize the demultiplexer part of the RTCRtpReceiver to forward frames with the payload type identical to `parameters.payloadType` to this decoder. +When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument, and initialize the demultiplexer part of the RTCRtpReceiver that follows the transform step to forward frames with the payload type identical to `parameters.payloadType` to this decoder.
+

+setEncodingCodec may be the same thing as setting encoder in RTCRtpEncodingParameters, but I'm not sure of that; the +two may have different effects on the packetizer. Also the "codec" is an RTCRtpCodec, while this uses RTCRtpCodecParameters +(which includes the payload type). +
+setEncodingParameters() does not affect decoding, so addDecodingCodec is needed. +

+ ## Extension operation ## {#operation} ### Media type negotiation ### {#sdp-negotiation} From c0bf9648ee5a396d71ac149d5c384ef8ff140a9e Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 29 Aug 2023 15:05:42 +0200 Subject: [PATCH 14/42] Updated sdp-explainer.md with new code++ --- sdp-explainer.md | 72 +++++++++++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 31 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index d389c35..844263b 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -8,6 +8,10 @@ This creates a problem, in that when an encoded transform is applied on the send This problem is even more acute when the Encoded Transform is used to add support for payload types not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to demultiplex the incoming packets correctly and route them to the right decoder. +For example, it's been proposed to add [Lyra](https://github.com/google/lyra) to WebRTC using an implementation in WASM +- a working example using SDP munging can be found on the +[Meetecho blog](https://www.meetecho.com/blog/playing-with-lyra/). + # Requirements for an SDP negotiation API The following things need to be available on such an API: 1. Before SDP negotiation, the application must be able to specify one or more new media types that one wants to negotiate for. As a point of illustration, this document uses the type "video/new-codec". @@ -52,7 +56,8 @@ When transforming frames, the transformer configured MUST, in addition to modify The packetizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. (This assumes that packetization is independent of FMTP) ## For receiving -The depacketizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. +The depacketizer will use the rules for the MIME type associated with the payload type. If the payload type is associated +with an RtpCodecCapability that has a packetizationMode parameter, that packetization mode will be used. The decoder can be configured to accept a given PT as indicating a given codec format by the new API call: ``` @@ -78,49 +83,54 @@ customCodec = { // At sender side pc.addSendCodecCapability('video', customCodec); -sender = pc.AddTrack(videotrack); -// Negotiate as usual -for (codec in sender.getParameters().codecs) { - if (codec.mimeType == “video/x-encrypted”) { - encryptedPT = codec.payloadType; - } else if (codec.mimeType == 'video/vp8') { - encodingCodec = codec; - } + +// ...after negotiation + +const {codecs} = sender.getParameters(); +const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); + +const worker = new Worker(`data:text/javascript,(${work.toString()})()`); +sender.transform = new RTCRtpScriptTransform(worker, {payloadType}); + +function work() { + onrtctransform = async ({transformer: {readable, writable, options}}) => + await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable); + + function transform(frame, controller) { + // transform chunk + let metadata = frame.metadata(); + encryptBody(frame); + metadata = frame.metadata(); + metadata.pt = options.payloadType; + frame.setMetadata(metadata); + controller.enqueue(frame); + } } -sender.setEncodingCodec(encodingCodec); -(readable, writable) = sender.getEncodedStreams(); - -readable.pipeThrough(new TransformStream( - transform: (frame) => { - encryptBody(frame); - metadata = frame.metadata(); - metadata.pt = encryptedPT; - frame.setMetadata(metadata); - writable.write(frame); - } -}).pipeTo(writable); // At receiver side. const decryptedPT = 208; // Can be negotiated PT or locally-valid pc.addReceiveCodecCapability('video', customCodec); pc.ontrack = (receiver) => { - for (codec in receiver.getParameters().codecs) { - if (codec.mimeType == “application/x-encrypted”) { - encryptedPT = codec.payloadType; - } - } + const {codecs} = sender.getParameters(); + const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); + receiver.addDecodingCodec({mimeType: 'video/vp8', payloadType: decryptedPT}); - (readable, writable) = receiver.getEncodedStreams(); - readable.pipeThrough(new TransformStream( - transform: (frame) => { + const worker = new Worker(`data:text/javascript,(${work.toString()})()`); + receiver.transform = new RTCRtpScriptTransform(worker, {payloadType}); + + function work() { + transform = new TransformStream( + transform: (frame, controller) => { if (frame.metadata().payloadType != encryptedPT) { continue; // Ignore all frames for the wrong payload. } decryptBody(frame); metadata = frame.metadata(); metadata.payloadType = decryptedPT; - writable.write(frame); + controller.enqueue(frame); } - }).pipeTo(writable); + }); + onrtctransform = async({transformer: {readable, writable, options}}) => + await readable.pipeThrough(transform).pipeTo(writable); }; ``` From ef896ab3fb63c90635c84510824688be5da8342c Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 6 Oct 2023 14:54:42 +0200 Subject: [PATCH 15/42] Introduce setPacketizer This replaces the "packetizer" field of the custom codec description. --- index.bs | 51 +++++++++++++++++++++++++-------------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/index.bs b/index.bs index db7f898..5c6f0a1 100644 --- a/index.bs +++ b/index.bs @@ -86,7 +86,6 @@ initially empty. 1. Let 'capability' be the "capability" argument 1. If 'capability' is identical to the capability describing a previously configured codec, throw an IllegalModification error. -1. If 'packetizationMode' is defined, and is not a known codec, throw a SyntaxError DOMException. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot. #### AddReceiveCodecCapability #### {#add-receive-codec-capability} @@ -94,23 +93,8 @@ initially empty. 1. Let 'capability' be the "capability" argument 1. If 'capability' is identical to the capability describing a previously configured codec, throw an IllegalModification error. -1. If 'packetizationMode' is defined, and is not a known codec, throw a SyntaxError DOMException. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot. -## RTCRtpCodecCapability extensions ## {#rtc-rtp-codec-capability-extensions} - -
-partial dictionary RTCRtpCodecCapability {
-  DOMString packetizationMode;
-};
-
- -### Attributes ### {#rtc-rtp-codec-capability-attributes} -#### packetizationMode #### {#packetization-mode} -Defines the packetization mode to use for the frame being packetized or depacketized. MUST be the mimeType from -the capability describing a codec in the [=list of implemented send codecs=] (for packetizing) or the -[=list of implemented receive codecs=] (for depacketizing). - ## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions} 
 
@@ -119,32 +103,47 @@ typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
 // New methods and attributes for RTCRtpSender and RTCRtpReceiver
 partial interface RTCRtpSender {
     attribute RTCRtpTransform? transform;
-    undefined setEncodingCodec(RTCRtpCodecParameters parameters);
+    undefined setPacketizer(octet payloadType, RTCRtpCodec packetizer);
 };
 
 partial interface RTCRtpReceiver {
     attribute RTCRtpTransform? transform;
-    undefined addDecodingCodec(RTCRtpCodecParameters parameters);};
+    undefined setDepacketizer(octet payloadType, RTCRtpCodec depacketizer);
+    undefined addDecodingCodec(octet payloadType, RTCRtpCodec parameters);
+};
### New methods ### {#sender-receiver-new-methods}
-
setEncodingCodec
+
setPacketizer
+
+When called, indicate that the packetizer should behave as if frames +enqueued on the sender with the indicated payload type should be packetized +according to the rules of the codec indicated.
+It is up to the transform to ensure that all data and metadata conforms +to the format required by the packetizer; the sender may drop frames that +do not conform. {{RTCRtpOutboundRtpStreamStats/framesSent}} is not incremented +when a frame is dropped. +
+
setDepacketizer
-When called, switch the encoder in the RTCRtpSender to produce frames in the format indicated by the `parameters` argument. +When called, indicate that incoming packets with the `payloadType` value in +their PT field should be routed to a depacketizer that will perform +depacketization according to the rules of `depacketizer`.
addDecodingCodec
-When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument, and initialize the demultiplexer part of the RTCRtpReceiver that follows the transform step to forward frames with the payload type identical to `parameters.payloadType` to this decoder. +When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument.

-setEncodingCodec may be the same thing as setting encoder in RTCRtpEncodingParameters, but I'm not sure of that; the -two may have different effects on the packetizer. Also the "codec" is an RTCRtpCodec, while this uses RTCRtpCodecParameters -(which includes the payload type). -
-setEncodingParameters() does not affect decoding, so addDecodingCodec is needed. +The expected mode of operation is that the sender transform will transform +the frame and change its `payloadType` to the negotiated value for the +transformed content. The receiver will receive the `payloadType`, transform +it and set the `payloadType` to the value that corresponds to the format +of the decoded frame, thus ensuring that it is decoded using the correct +decoder.

## Extension operation ## {#operation} From d582ea5178b2bc3faa2ef1ac3fa2c12577ea7a5e Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 6 Oct 2023 14:55:45 +0200 Subject: [PATCH 16/42] Minifix --- index.bs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/index.bs b/index.bs index 5c6f0a1..2361853 100644 --- a/index.bs +++ b/index.bs @@ -122,7 +122,7 @@ enqueued on the sender with the indicated payload type should be packetized according to the rules of the codec indicated.
It is up to the transform to ensure that all data and metadata conforms to the format required by the packetizer; the sender may drop frames that -do not conform. {{RTCRtpOutboundRtpStreamStats/framesSent}} is not incremented +do not conform. {{RTCOutboundRtpStreamStats/framesSent}} is not incremented when a frame is dropped.
setDepacketizer
@@ -140,8 +140,8 @@ When called, initialize a decoder in the RTCRtpReceiver to handle the format ind

The expected mode of operation is that the sender transform will transform the frame and change its `payloadType` to the negotiated value for the -transformed content. The receiver will receive the `payloadType`, transform -it and set the `payloadType` to the value that corresponds to the format +transformed content. The receiver transform will receive the `payloadType`, +transform it, and set the `payloadType` to the value that corresponds to the format of the decoded frame, thus ensuring that it is decoded using the correct decoder.

From a35b9fd6c906c7622d011d975c533187db606636 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 6 Oct 2023 15:18:31 +0200 Subject: [PATCH 17/42] Minifix --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index 2361853..353f169 100644 --- a/index.bs +++ b/index.bs @@ -75,7 +75,7 @@ partial interface RTCPeerConnection { In addition, add two new internal slots to the RTCPeerConnection, called -{{RTCPeerConnection/[[CustomSendCodecs]]}} +{{RTCPeerConnection/[[CustomSendCodecs]]}} and {{RTCPeerConnection/[[CustomReceiveCodecs]]}}, initially empty. From d433822f1e94e816f562e020635a48192572dbd7 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Sun, 8 Oct 2023 13:37:04 +0200 Subject: [PATCH 18/42] Apply suggestions from code review <dfn> fixes Co-authored-by: Dominique Hazael-Massieux --- index.bs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.bs b/index.bs index 353f169..b51f7fc 100644 --- a/index.bs +++ b/index.bs @@ -75,8 +75,8 @@ partial interface RTCPeerConnection { In addition, add two new internal slots to the RTCPeerConnection, called -{{RTCPeerConnection/[[CustomSendCodecs]]}} -and {{RTCPeerConnection/[[CustomReceiveCodecs]]}}, +\[[CustomSendCodecs]] +and \[[CustomReceiveCodecs]], initially empty. ### Methods ### {#rtcpeerconnection-methods} From 713a7719899b9b9ed8b9a43ba89b19ba9cb5fc81 Mon Sep 17 00:00:00 2001 From: Dominique Hazael-Massieux Date: Mon, 9 Oct 2023 11:12:59 +0200 Subject: [PATCH 19/42] Fix a couple of other references --- index.bs | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/index.bs b/index.bs index b51f7fc..84a7636 100644 --- a/index.bs +++ b/index.bs @@ -154,10 +154,10 @@ The following modifications are applied to the session negotation procedure: 1. The {{RTCPeerConnection/[[CustomSendCodecs]]}} is included in the list of codecs which the user agent is currently capable of sending; the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. -1. In the [=final steps to create an offer=], step 2.2 is changed to filter the list of codecs against - the union of the [=RTCRtpSender/list of implemented send codecs=] and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the [=RTCRtpReceiver/list of implemented receive codecs=] and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot. +1. In the [$final steps to create an offer$], step 2.2 is changed to filter the list of codecs against + the union of the [=RTCRtpSender/list of implemented send codecs=] and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the [=list of implemented receive codecs=] and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot. -1. In the [=final steps to create an answer=], the same modification is done to the filtering in steps 1.2 and 1.3. +1. In the [$final steps to create an answer$], the same modification is done to the filtering in steps 1.2 and 1.3. From 5f483bc21b0ac91bc1d33fcb83bfcea61f580820 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 13 Oct 2023 14:09:01 +0200 Subject: [PATCH 20/42] Explainer update --- sdp-explainer.md | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 844263b..64f27fd 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -24,11 +24,10 @@ The following things need to be available on such an API: # API description ## Codec description -For codec description, we reuse the dictionary RTCRtpCodecCapability, but add a DOMString that identifies the packetization mode. +For codec description, we reuse the dictionary RTCRtpCodecCapability. The requirements on the parameters are: - either mimetype or fmtp parameters must be different from any existing capability -- if the mimetype is not a known codec, the packetizationMode member MUST be given, and be the mimetype of a known codec. When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getCapabilities(). @@ -45,23 +44,15 @@ NOTE: The codecs will not show up on the static sender/receiver getCapabilities ## For sending The RTCRtpSender’s encoder (if present) will be configured to use a specific codec from CodecCapabilities by a new call: ``` -sender.setEncodingCodec(RTCCodecParameters parameters) +sender.setPacketizer(payloadType, RTCCodecParameters parameters) ``` -This sets the MIME type of the codec to encode to, and the payload type that will be put on frames produced by that encoder. This codec must be one supported by the platform (not the “novel” codec), and the PT does not need to be one negotiated in the SDP offer/answer. - -When configuring the transform post negotiation, the app MUST retrieve the PTs negotiated for the connection, and identify the PT for the custom codec. - -When transforming frames, the transformer configured MUST, in addition to modifying the payload, modify the metadata to have the negotiated payload type for the custom codec. - -The packetizer will use the rules for the MIME type configured, or the MIME type on the packetizationMode if configured. (This assumes that packetization is independent of FMTP) +This tells the sender that when packetizing a frame for transmission, it should use the rules described in the documentation for the codec indicated. ## For receiving -The depacketizer will use the rules for the MIME type associated with the payload type. If the payload type is associated -with an RtpCodecCapability that has a packetizationMode parameter, that packetization mode will be used. - -The decoder can be configured to accept a given PT as indicating a given codec format by the new API call: +The depacketizer will use the rules for the MIME type associated with the payload type. +If the MIME type associated with the payload type has been added using addReceiveCodecCapability, the decoder must be configured to accept a given PT as indicating a given codec format by the new API call: ``` -receiver.addDecodingCodec(CodecParameters parameters) +receiver.setDepacketizer(payloadType, CodecParameters parameters) ``` This does not alter the handling of any otherwise-configured PT, but adds a handler for this specific PT. @@ -70,6 +61,7 @@ On seeing a custom codec in the PT for an incoming frame, if the frame is to be ## Existing APIs that will be used together with the new APIs - Basic establishing of EncodedTransform - getParameters() to get results of codec negotiation +- encoded frame SetMetadata, to set the payload type for processed frames # Example code (This is incomplete) @@ -78,7 +70,6 @@ customCodec = { mimeType: “video/x-encrypted”, clockRate: 90000, fmtp = “encapsulated-codec=vp8”, - packetizationMode = “video/vp8”, }; // At sender side @@ -88,6 +79,7 @@ pc.addSendCodecCapability('video', customCodec); const {codecs} = sender.getParameters(); const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); +sender.setPacketizer(encryptingCodec.payloadType, {mimeType: "video/vp8"}); const worker = new Worker(`data:text/javascript,(${work.toString()})()`); sender.transform = new RTCRtpScriptTransform(worker, {payloadType}); @@ -112,8 +104,8 @@ const decryptedPT = 208; // Can be negotiated PT or locally-valid pc.addReceiveCodecCapability('video', customCodec); pc.ontrack = (receiver) => { const {codecs} = sender.getParameters(); - const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); - + const encryptingCodec = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); + receiver.setDepacketizer(encryptingCodec.payloadType, {mimeType: "video/vp8"}); receiver.addDecodingCodec({mimeType: 'video/vp8', payloadType: decryptedPT}); const worker = new Worker(`data:text/javascript,(${work.toString()})()`); receiver.transform = new RTCRtpScriptTransform(worker, {payloadType}); @@ -134,3 +126,17 @@ pc.ontrack = (receiver) => { await readable.pipeThrough(transform).pipeTo(writable); }; ``` + +# Frequently asked questions + +1. Q: Why is the SDP negotiation interface on PeerConnection and not on Sender/Receiver or on the Transform? + A: WebRTC connections go through three phases: SDP negotiation, ICE/DTLS transport setup, and sending/receiving media. In the SDP negotiation phase, the ICE/DTLS transport does not exist. + +1. Q: Why are the packetizer controllers on the Sender/Receiver and not on the Transform? + A: The Transform API is agnostic as to what direction the packets are destined for. The packetizer control only makes sense if the destination for the frames is something that will packetize the frames for sending across an ICE transport. + +1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that? + A: Use multiple payload types. Each will be assigned a payload type. Mark each frame with the payload type they need to be packetized as. + +1. Q: What is the relationship between this proposal and the IETF standard for SFrame? + A: This proposal is intended to make it possible to implement the IETF standard for SFrame in Javascript, with only the packetizer/depacketizer being updated to support the SFrame packetization rules. It is also intended to make it possible to perform other forms of transform, including the ones that are presently deployed in the field, while marking the SDP with a truthful statement of its content. From deb9866165185d6c579d4de892aad13a9e81c063 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 13 Oct 2023 14:19:37 +0200 Subject: [PATCH 21/42] Fix md formatting --- sdp-explainer.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sdp-explainer.md b/sdp-explainer.md index 64f27fd..5d4462c 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -130,13 +130,17 @@ pc.ontrack = (receiver) => { # Frequently asked questions 1. Q: Why is the SDP negotiation interface on PeerConnection and not on Sender/Receiver or on the Transform? + A: WebRTC connections go through three phases: SDP negotiation, ICE/DTLS transport setup, and sending/receiving media. In the SDP negotiation phase, the ICE/DTLS transport does not exist. 1. Q: Why are the packetizer controllers on the Sender/Receiver and not on the Transform? + A: The Transform API is agnostic as to what direction the packets are destined for. The packetizer control only makes sense if the destination for the frames is something that will packetize the frames for sending across an ICE transport. 1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that? + A: Use multiple payload types. Each will be assigned a payload type. Mark each frame with the payload type they need to be packetized as. 1. Q: What is the relationship between this proposal and the IETF standard for SFrame? + A: This proposal is intended to make it possible to implement the IETF standard for SFrame in Javascript, with only the packetizer/depacketizer being updated to support the SFrame packetization rules. It is also intended to make it possible to perform other forms of transform, including the ones that are presently deployed in the field, while marking the SDP with a truthful statement of its content. From a02ac43e4447f0a7f0786d0fabcbbf42b29e1824 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Mon, 13 Nov 2023 22:54:22 +0100 Subject: [PATCH 22/42] Explain in terms of transceiver --- sdp-explainer.md | 45 ++++++++++++++++++--------------------------- 1 file changed, 18 insertions(+), 27 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 5d4462c..e5466df 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -33,30 +33,30 @@ When a codec capability is added, the SDP machinery will negotiate these codecs ## For SDP negotiation -``` -pc.addSendCodecCapability(DOMString kind, CodecCapability capability) -pc.addReceiveCodecCapability(DOMString kind, CodecCapability capability) -``` -These calls will add to the lists of codecs being negotiated in SDP, and returned by the calls to getParameters. (Given the rules for generating SDP, the effect on sendonly/recvonly/sendrecv sections in the SDP will be different. Read those rules with care.) +SDP negotiation is inherently an SDP property, and the proper target for that is therefore the RTCRtpTransceiver object. -NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. +The transceiver will always be available before an offer or answer is created (either because of AddTrack or AddTransceiver on the sender side, +or in the ontrack event on the side that receives the offer). At that time, the additional codecs to negotiate must be set: -## For sending -The RTCRtpSender’s encoder (if present) will be configured to use a specific codec from CodecCapabilities by a new call: ``` -sender.setPacketizer(payloadType, RTCCodecParameters parameters) +transceiver.addCodec(RTCRtpCodec codec, RTCRtpCodec packetizationMode) ``` -This tells the sender that when packetizing a frame for transmission, it should use the rules described in the documentation for the codec indicated. +This will add the described codec to the list of codecs to be offered in createOffer/createAnswer. On successful negotiation +of the codec, it will appear in the negotiated codec list returned from setParameters(). + +Furthermore, the call will instruct the sender to packetize, and the receiver to depacketize, in accordance with the rules for the +codec being given as the packetizationMode argument. + +Because it is sometimes inconvenient to intercept every call that creates transceivers, a convenience method is offered on the PC level: -## For receiving -The depacketizer will use the rules for the MIME type associated with the payload type. -If the MIME type associated with the payload type has been added using addReceiveCodecCapability, the decoder must be configured to accept a given PT as indicating a given codec format by the new API call: ``` -receiver.setDepacketizer(payloadType, CodecParameters parameters) +pc.addCodecCapability(DOMString kind, RTCRtpCodec codec, RTCRtpCodec packetizationMode) ``` -This does not alter the handling of any otherwise-configured PT, but adds a handler for this specific PT. +These calls will act as if the addCodec() call had been invoked on every transceiver created of the associated "kind". + +NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. + -On seeing a custom codec in the PT for an incoming frame, if the frame is to be delivered to the corresponding decoder, the encoded frame transform MUST, in addition to transforming the payload, set the PT for the frame to a PT that is understood by the decoder - either by being negotiated or by having been added by AddDecodingCodec. ## Existing APIs that will be used together with the new APIs - Basic establishing of EncodedTransform @@ -73,13 +73,12 @@ customCodec = { }; // At sender side -pc.addSendCodecCapability('video', customCodec); +pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"}); // ...after negotiation const {codecs} = sender.getParameters(); const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); -sender.setPacketizer(encryptingCodec.payloadType, {mimeType: "video/vp8"}); const worker = new Worker(`data:text/javascript,(${work.toString()})()`); sender.transform = new RTCRtpScriptTransform(worker, {payloadType}); @@ -101,7 +100,7 @@ function work() { // At receiver side. const decryptedPT = 208; // Can be negotiated PT or locally-valid -pc.addReceiveCodecCapability('video', customCodec); +pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"}); pc.ontrack = (receiver) => { const {codecs} = sender.getParameters(); const encryptingCodec = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); @@ -129,14 +128,6 @@ pc.ontrack = (receiver) => { # Frequently asked questions -1. Q: Why is the SDP negotiation interface on PeerConnection and not on Sender/Receiver or on the Transform? - - A: WebRTC connections go through three phases: SDP negotiation, ICE/DTLS transport setup, and sending/receiving media. In the SDP negotiation phase, the ICE/DTLS transport does not exist. - -1. Q: Why are the packetizer controllers on the Sender/Receiver and not on the Transform? - - A: The Transform API is agnostic as to what direction the packets are destined for. The packetizer control only makes sense if the destination for the frames is something that will packetize the frames for sending across an ICE transport. - 1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that? A: Use multiple payload types. Each will be assigned a payload type. Mark each frame with the payload type they need to be packetized as. From 5cecfe3fd24983ad4f5d45fdde114c7e3e718815 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 14 Nov 2023 15:28:12 +0100 Subject: [PATCH 23/42] Sframe -> SFrame Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index e5466df..1f485e9 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -2,7 +2,7 @@ The original definition of encoded transform did not consider negotation; the frames on the "transformed" side went out stamped with the payload type of the frame that came in on the "non-transformed" side (and vice versa for the receiver). -This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire do not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging Sframe encrypted media between two endpoints from different application providers, or when exchanging Sframe encrypted content via an SFU that expects to be able to decode or modify the media. +This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire may not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging SFrame encrypted media between two endpoints from different application providers, or when exchanging SFrame encrypted content via an SFU that expects to be able to decode or modify the media. (The latter is exactly what Sframe is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.) From f3be9ddfa9c57312949600234e9daf0b2b0201a5 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 16 Nov 2023 07:43:16 +0000 Subject: [PATCH 24/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 1f485e9..8dc1345 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -6,7 +6,7 @@ This creates a problem, in that when an encoded transform is applied on the send (The latter is exactly what Sframe is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.) -This problem is even more acute when the Encoded Transform is used to add support for payload types not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to demultiplex the incoming packets correctly and route them to the right decoder. +This problem is even more acute when the Encoded Transform is used to add support for codecs not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to associate the payload type of incoming packets with the right decoder. For example, it's been proposed to add [Lyra](https://github.com/google/lyra) to WebRTC using an implementation in WASM - a working example using SDP munging can be found on the From abd0235764c63f3662c7b4d9156947bf9f0815b7 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 17 Nov 2023 14:17:14 +0000 Subject: [PATCH 25/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 8dc1345..03698c9 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -29,7 +29,7 @@ For codec description, we reuse the dictionary RTCRtpCodecCapability. The requirements on the parameters are: - either mimetype or fmtp parameters must be different from any existing capability -When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getCapabilities(). +When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getParameters(). ## For SDP negotiation From 4e34b4c30796412853988a8db8570db96fe88ac6 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 17 Nov 2023 14:18:36 +0000 Subject: [PATCH 26/42] Update sdp-explainer.md Co-authored-by: Jan-Ivar Bruaroey --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 03698c9..4d2c806 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -101,7 +101,7 @@ function work() { // At receiver side. const decryptedPT = 208; // Can be negotiated PT or locally-valid pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"}); -pc.ontrack = (receiver) => { +pc.ontrack = ({receiver}) => { const {codecs} = sender.getParameters(); const encryptingCodec = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); receiver.setDepacketizer(encryptingCodec.payloadType, {mimeType: "video/vp8"}); From 97f16acd3875fcf26bb5de0d49c97eb968a16269 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 17 Nov 2023 14:19:01 +0000 Subject: [PATCH 27/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 4d2c806..f4564f7 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -54,7 +54,7 @@ pc.addCodecCapability(DOMString kind, RTCRtpCodec codec, RTCRtpCodec packetizati ``` These calls will act as if the addCodec() call had been invoked on every transceiver created of the associated "kind". -NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in getParameters(), so they’re available for selection or deselection. +NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in RTCRtp{Sender,Receiver}.getParameters(), so they’re available to the RTCRtpSender for selection or deselection. From d779d392dd2864b385c542ee4cff0ee63119b02a Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 21 Nov 2023 13:31:12 +0100 Subject: [PATCH 28/42] Fix example code --- sdp-explainer.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index f4564f7..0104bcb 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -110,19 +110,20 @@ pc.ontrack = ({receiver}) => { receiver.transform = new RTCRtpScriptTransform(worker, {payloadType}); function work() { - transform = new TransformStream( + transform = new TransformStream({ transform: (frame, controller) => { if (frame.metadata().payloadType != encryptedPT) { - continue; // Ignore all frames for the wrong payload. + return; // Ignore all frames for the wrong payload. } decryptBody(frame); metadata = frame.metadata(); metadata.payloadType = decryptedPT; controller.enqueue(frame); } - }); - onrtctransform = async({transformer: {readable, writable, options}}) => - await readable.pipeThrough(transform).pipeTo(writable); + }); + onrtctransform = async({transformer: {readable, writable, options}}) => + await readable.pipeThrough(transform).pipeTo(writable); + } }; ``` From b44d67fcdba561fb33515f3c64e3233a25cc451d Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 15 Dec 2023 13:27:26 +0100 Subject: [PATCH 29/42] Add "both alternatives" API description This update adds both API alternatives: data-on-transform and data-on-transceiver. It also adds the "filtering" function for transforms. --- sdp-explainer.md | 82 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 61 insertions(+), 21 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 0104bcb..5497766 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -6,12 +6,13 @@ This creates a problem, in that when an encoded transform is applied on the send (The latter is exactly what Sframe is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.) -This problem is even more acute when the Encoded Transform is used to add support for codecs not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to associate the payload type of incoming packets with the right decoder. +This problem is even more acute if an interface resembling RTCRtpScriptTransform is used to add support for codecs not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to associate the payload type of incoming packets with the right decoder. -For example, it's been proposed to add [Lyra](https://github.com/google/lyra) to WebRTC using an implementation in WASM -- a working example using SDP munging can be found on the +For example, it's been proposed to add [Lyra](https://github.com/google/lyra) to WebRTC using an implementation in WASM; a working example using SDP munging can be found on the [Meetecho blog](https://www.meetecho.com/blog/playing-with-lyra/). +However, this API proposal does not directly address that use case at the moment. + # Requirements for an SDP negotiation API The following things need to be available on such an API: 1. Before SDP negotiation, the application must be able to specify one or more new media types that one wants to negotiate for. As a point of illustration, this document uses the type "video/new-codec". @@ -24,22 +25,47 @@ The following things need to be available on such an API: # API description ## Codec description -For codec description, we reuse the dictionary RTCRtpCodecCapability. +For codec description, we reuse the dictionary RTCRtpCodecCapability, but add a new field describing the +packetization mode to be used. The requirements on the parameters are: - either mimetype or fmtp parameters must be different from any existing capability +- the packetization mode must identify a mode known to the UA. When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getParameters(). +## Describing the input and output codecs of transforms + +We extend the RTCRTPScriptTransform object's constructor with a fourth argument of type CodecInformation, with the following IDL definition: + +``` +dictionary CodecInformation { + sequence inputCodecs; + sequence outputCodecs; + bool acceptOnlyInputCodecs = false; +} +``` +The inputCodecs member describe the media types the transform is prepared to process. Any frame of a format +not listed will be passed to the output of the transform without modification. + +The outputCodecs describes the media types the transform may produce. + +NOTE: The inputCodecs has two purposes in the "Transform proposal" below - it gives codecs to +negotiate in the SDP, and it serves to filter the frame types that the transform will process. -## For SDP negotiation +In order to be able to use the filtering function, the "acceptOnlyInputCodecs" has to be set to true; +if it is false, all frames are delivered to the transform. + +The next section has two versions - the Transceiver proposal and the Transform proposal. + +## For SDP negotiation - Transceiver proposal SDP negotiation is inherently an SDP property, and the proper target for that is therefore the RTCRtpTransceiver object. The transceiver will always be available before an offer or answer is created (either because of AddTrack or AddTransceiver on the sender side, or in the ontrack event on the side that receives the offer). At that time, the additional codecs to negotiate must be set: ``` -transceiver.addCodec(RTCRtpCodec codec, RTCRtpCodec packetizationMode) +transceiver.addCodec(RTCRtpCodecWithPacketization codec, RTCRtpTransceiverDirection direction = "sendrecv") ``` This will add the described codec to the list of codecs to be offered in createOffer/createAnswer. On successful negotiation of the codec, it will appear in the negotiated codec list returned from setParameters(). @@ -50,35 +76,51 @@ codec being given as the packetizationMode argument. Because it is sometimes inconvenient to intercept every call that creates transceivers, a convenience method is offered on the PC level: ``` -pc.addCodecCapability(DOMString kind, RTCRtpCodec codec, RTCRtpCodec packetizationMode) +pc.addCodecCapability(RTCRtpCodecWithPacketization codec, RTCRtpTransceiverDirection direction = "sendrecv") ``` These calls will act as if the addCodec() call had been invoked on every transceiver created of the associated "kind". -NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in RTCRtp{Sender,Receiver}.getParameters(), so they’re available to the RTCRtpSender for selection or deselection. +NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections or, when using transceiver-level addCodec, for different transceivers. They will show up in the list of codecs in RTCRtp{Sender,Receiver}.getParameters(), so they’re available to the RTCRtpSender for selection or deselection using setCodecPreferences(). + +When the PeerConnection generates an offer or an answer, the codecs added with addCodec() will be added to the list of codecs generated for that transceiver. If "direction" is sendrecv or recvonly, the selected +payload types will also be added to the list of receive codecs on the m= line; if "direction" is "sendonly", the selected payload types will not be added to the list. + +Note: It is the application's job to ensure that the transform and the negotiation agree on which +codecs are being processed. + +## For SDP negotiation - Transform proposal + +When the PeerConnection generates an offer or an answer: + +* If a transform is set on the sender, the process for generating an offer or answer will add the codecs + listed in the transform's outputCodecs to the list of codecs available for sending. +* If a transform is set on the receiver, the process for generating an offer or answer will add the codecs + listed in the transform's inputCodecs to the list of codecs available for receiving. +When the transform attribute of a sender or receiver is changed, and the relevant codec list changes, the "negotiationneeded" event fires. ## Existing APIs that will be used together with the new APIs - Basic establishing of EncodedTransform - getParameters() to get results of codec negotiation - encoded frame SetMetadata, to set the payload type for processed frames +- setCodecPreferences, to say which codecs (old or new) are preferred for reception # Example code -(This is incomplete) +(This is based on the Transceiver API proposal) ``` customCodec = { mimeType: “video/x-encrypted”, clockRate: 90000, - fmtp = “encapsulated-codec=vp8”, + packetizationMode: "video/vp8", }; // At sender side -pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"}); +pc.addCodecCapability(customCodec); // ...after negotiation const {codecs} = sender.getParameters(); -const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); const worker = new Worker(`data:text/javascript,(${work.toString()})()`); sender.transform = new RTCRtpScriptTransform(worker, {payloadType}); @@ -92,7 +134,7 @@ function work() { let metadata = frame.metadata(); encryptBody(frame); metadata = frame.metadata(); - metadata.pt = options.payloadType; + metadata.mediaType = "video/x-encrypted" frame.setMetadata(metadata); controller.enqueue(frame); } @@ -100,24 +142,22 @@ function work() { // At receiver side. const decryptedPT = 208; // Can be negotiated PT or locally-valid -pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"}); +pc.addCodecCapability('video', customCodec); pc.ontrack = ({receiver}) => { const {codecs} = sender.getParameters(); - const encryptingCodec = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted"); - receiver.setDepacketizer(encryptingCodec.payloadType, {mimeType: "video/vp8"}); receiver.addDecodingCodec({mimeType: 'video/vp8', payloadType: decryptedPT}); const worker = new Worker(`data:text/javascript,(${work.toString()})()`); - receiver.transform = new RTCRtpScriptTransform(worker, {payloadType}); + receiver.transform = new RTCRtpScriptTransform(worker, null, null, + {inputCodecs: [customCodec], acceptOnlyInputCodecs = True}); function work() { transform = new TransformStream({ transform: (frame, controller) => { - if (frame.metadata().payloadType != encryptedPT) { - return; // Ignore all frames for the wrong payload. - } + // We know that only encrypted-frames will get to this point. decryptBody(frame); metadata = frame.metadata(); - metadata.payloadType = decryptedPT; + metadata.mimeType = 'video/vp8'; + frame.setMetadata(metadata); controller.enqueue(frame); } }); From 295884423d6ff94301be5cb0ed2b6ecc40806683 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 12 Jan 2024 11:06:10 +0100 Subject: [PATCH 30/42] Update sdp-explainer.md Separate the two proposals better Co-authored-by: Jan-Ivar Bruaroey --- sdp-explainer.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 5497766..d00eb05 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -106,9 +106,8 @@ When the transform attribute of a sender or receiver is changed, and the relevan - encoded frame SetMetadata, to set the payload type for processed frames - setCodecPreferences, to say which codecs (old or new) are preferred for reception -# Example code -(This is based on the Transceiver API proposal) -``` +# Example code based on the Transceiver API proposal +```js customCodec = { mimeType: “video/x-encrypted”, clockRate: 90000, From 78fbfbc4f80704efce4fe7ec81a77b74d5ef2057 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Fri, 12 Jan 2024 11:06:41 +0100 Subject: [PATCH 31/42] Update sdp-explainer.md Separate the two proposals better Co-authored-by: Jan-Ivar Bruaroey --- sdp-explainer.md | 49 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/sdp-explainer.md b/sdp-explainer.md index d00eb05..9f05dc4 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -166,6 +166,55 @@ pc.ontrack = ({receiver}) => { }; ``` +# Example code based on the Transform API proposal + +```js +const worker = new Worker(`data:text/javascript,(${work.toString()})()`); + +// At sender side. +const sender = pc.addTrack(track); +const {codecs} = RTCRtpSender.getCapabilities(); +const vp8 = codecs.find(({mimeType}) => mimeType == "video/vp8"); +sender.transform = new RTCRtpScriptTransform(worker, { + inputCodecs: [vp8], + outputCodecs: [{mimeType: “video/x-encrypted”}] +}); + +// At receiver side. +pc.ontrack = ({receiver}) => { + const {codecs} = receiver.getParameters(); + const customCodec = codecs.find(({mimeType}) => mimeType == "video/x-encrypted"); + if (customCodec) { + receiver.transform = new RTCRtpScriptTransform(worker, { + inputCodecs: [customCodec], + outputCodecs: [{mimeType: "video/vp8"}] + }); + } +} + +// Same worker can handle both sides. +function work() { + onrtctransform = async ({transformer: {readable, writable}, {inputCodecs, outputCodecs}}) => { + const [outputCodec] = outputCodecs; + await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable); + function transform(frame, controller) { + // transform chunk + let metadata = frame.metadata(); + const inputCodec = inputCodecs.find((mimeType) => mimeType == metadata.mediaType); + + if (inputCodec && outputCodec.mimeType == "video/x-encrypted") { + encryptBody(frame, inputCodec); + metadata.mediaType = outputCodec.mimeType; + frame.setMetadata(metadata); + } else if (inputCodec.mimeType == "video/x-encrypted") { + decryptBody(frame, outputCodec); + metadata.mediaType = outputCodec.mimeType; + frame.setMetadata(metadata); + } + controller.enqueue(frame); + } + } +} # Frequently asked questions 1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that? From a975401aa8ab0a0775a28319d12be9c200c71888 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 25 Jan 2024 11:56:12 +0100 Subject: [PATCH 32/42] Update sdp-explainer.md Co-authored-by: Jan-Ivar Bruaroey --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 9f05dc4..5bc88b6 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -1,6 +1,6 @@ # SDP negotiation in Encoded Transform -The original definition of encoded transform did not consider negotation; the frames on the "transformed" side went out stamped with the payload type of the frame that came in on the "non-transformed" side (and vice versa for the receiver). +The original definition of encoded transform did not consider negotiation; the frames on the "transformed" side went out stamped with the payload type of the frame that came in on the "non-transformed" side (and vice versa for the receiver). This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire may not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging SFrame encrypted media between two endpoints from different application providers, or when exchanging SFrame encrypted content via an SFU that expects to be able to decode or modify the media. From 8aa1533aa6a1a9b30f71802444cd98ef3d3a3879 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 25 Jan 2024 11:56:23 +0100 Subject: [PATCH 33/42] Update index.bs Co-authored-by: Jan-Ivar Bruaroey --- index.bs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.bs b/index.bs index 1315e8e..d5f7f5c 100644 --- a/index.bs +++ b/index.bs @@ -83,8 +83,8 @@ allow the declaration and negotiation of new media types. 
 // New methods for RTCPeerConnection
 partial interface RTCPeerConnection {
-   undefined AddSendCodecCapability(RTCRtpCodecCapability capability);
-   undefined AddReceiveCodecCapability(RTCRtpCodecCapability capability);
+   undefined addSendCodecCapability(RTCRtpCodecCapability capability);
+   undefined addReceiveCodecCapability(RTCRtpCodecCapability capability);
 };
From bd0c908d246856693bfa25ebebda4bbe2e143242 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 25 Jan 2024 11:56:34 +0100 Subject: [PATCH 34/42] Update index.bs Co-authored-by: Jan-Ivar Bruaroey --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index d5f7f5c..a040c30 100644 --- a/index.bs +++ b/index.bs @@ -99,7 +99,7 @@ initially empty. 1. Let 'capability' be the "capability" argument 1. If 'capability' is identical to the capability describing a previously configured codec, - throw an IllegalModification error. + throw an {{InvalidModificationError}} error. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot. #### AddReceiveCodecCapability #### {#add-receive-codec-capability} From bbc7f6b3c4b96dcc1f34846c357d0964e5c44d1a Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 25 Jan 2024 11:56:43 +0100 Subject: [PATCH 35/42] Update index.bs Co-authored-by: Jan-Ivar Bruaroey --- index.bs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.bs b/index.bs index a040c30..256d73e 100644 --- a/index.bs +++ b/index.bs @@ -106,7 +106,7 @@ initially empty. 1. Let 'capability' be the "capability" argument 1. If 'capability' is identical to the capability describing a previously configured codec, - throw an IllegalModification error. + throw an {{InvalidModificationError}} error. 1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot. ## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions} From 0ca754b353126ae7bd5c97e1a0b003aafca7c855 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 25 Jan 2024 11:57:02 +0100 Subject: [PATCH 36/42] Update sdp-explainer.md Co-authored-by: Philipp Hancke --- sdp-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 5bc88b6..56fce27 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -4,7 +4,7 @@ The original definition of encoded transform did not consider negotiation; the f This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire may not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging SFrame encrypted media between two endpoints from different application providers, or when exchanging SFrame encrypted content via an SFU that expects to be able to decode or modify the media. -(The latter is exactly what Sframe is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.) +(The latter is exactly what SFrame is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.) This problem is even more acute if an interface resembling RTCRtpScriptTransform is used to add support for codecs not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to associate the payload type of incoming packets with the right decoder. From ccf6ea8083f42baf75296ee50304e246925b3ad4 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 25 Jan 2024 12:02:20 +0100 Subject: [PATCH 37/42] Fix "pre" section end --- sdp-explainer.md | 1 + 1 file changed, 1 insertion(+) diff --git a/sdp-explainer.md b/sdp-explainer.md index 56fce27..8949cd6 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -215,6 +215,7 @@ function work() { } } } +``` # Frequently asked questions 1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that? From 6206a4676ff2ff9135bc02ca3d5a4e4144bedfc2 Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 27 Feb 2024 13:34:19 +0100 Subject: [PATCH 38/42] Update sdp explainer to remove transceiver alternative --- sdp-explainer.md | 101 ++++------------------------------------------- 1 file changed, 8 insertions(+), 93 deletions(-) diff --git a/sdp-explainer.md b/sdp-explainer.md index 8949cd6..a4fb3e4 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -36,6 +36,9 @@ When a codec capability is added, the SDP machinery will negotiate these codecs ## Describing the input and output codecs of transforms +We extend the RTCRtpCodecCapability object with a "packetizer" element, which identifies a media type with a packetizer +known to the platform. + We extend the RTCRTPScriptTransform object's constructor with a fourth argument of type CodecInformation, with the following IDL definition: ``` @@ -58,37 +61,7 @@ if it is false, all frames are delivered to the transform. The next section has two versions - the Transceiver proposal and the Transform proposal. -## For SDP negotiation - Transceiver proposal -SDP negotiation is inherently an SDP property, and the proper target for that is therefore the RTCRtpTransceiver object. - -The transceiver will always be available before an offer or answer is created (either because of AddTrack or AddTransceiver on the sender side, -or in the ontrack event on the side that receives the offer). At that time, the additional codecs to negotiate must be set: - -``` -transceiver.addCodec(RTCRtpCodecWithPacketization codec, RTCRtpTransceiverDirection direction = "sendrecv") -``` -This will add the described codec to the list of codecs to be offered in createOffer/createAnswer. On successful negotiation -of the codec, it will appear in the negotiated codec list returned from setParameters(). - -Furthermore, the call will instruct the sender to packetize, and the receiver to depacketize, in accordance with the rules for the -codec being given as the packetizationMode argument. - -Because it is sometimes inconvenient to intercept every call that creates transceivers, a convenience method is offered on the PC level: - -``` -pc.addCodecCapability(RTCRtpCodecWithPacketization codec, RTCRtpTransceiverDirection direction = "sendrecv") -``` -These calls will act as if the addCodec() call had been invoked on every transceiver created of the associated "kind". - -NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections or, when using transceiver-level addCodec, for different transceivers. They will show up in the list of codecs in RTCRtp{Sender,Receiver}.getParameters(), so they’re available to the RTCRtpSender for selection or deselection using setCodecPreferences(). - -When the PeerConnection generates an offer or an answer, the codecs added with addCodec() will be added to the list of codecs generated for that transceiver. If "direction" is sendrecv or recvonly, the selected -payload types will also be added to the list of receive codecs on the m= line; if "direction" is "sendonly", the selected payload types will not be added to the list. - -Note: It is the application's job to ensure that the transform and the negotiation agree on which -codecs are being processed. - -## For SDP negotiation - Transform proposal +## For SDP negotiation When the PeerConnection generates an offer or an answer: @@ -106,67 +79,8 @@ When the transform attribute of a sender or receiver is changed, and the relevan - encoded frame SetMetadata, to set the payload type for processed frames - setCodecPreferences, to say which codecs (old or new) are preferred for reception -# Example code based on the Transceiver API proposal -```js -customCodec = { - mimeType: “video/x-encrypted”, - clockRate: 90000, - packetizationMode: "video/vp8", -}; - -// At sender side -pc.addCodecCapability(customCodec); - -// ...after negotiation - -const {codecs} = sender.getParameters(); - -const worker = new Worker(`data:text/javascript,(${work.toString()})()`); -sender.transform = new RTCRtpScriptTransform(worker, {payloadType}); - -function work() { - onrtctransform = async ({transformer: {readable, writable, options}}) => - await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable); - - function transform(frame, controller) { - // transform chunk - let metadata = frame.metadata(); - encryptBody(frame); - metadata = frame.metadata(); - metadata.mediaType = "video/x-encrypted" - frame.setMetadata(metadata); - controller.enqueue(frame); - } -} - -// At receiver side. -const decryptedPT = 208; // Can be negotiated PT or locally-valid -pc.addCodecCapability('video', customCodec); -pc.ontrack = ({receiver}) => { - const {codecs} = sender.getParameters(); - receiver.addDecodingCodec({mimeType: 'video/vp8', payloadType: decryptedPT}); - const worker = new Worker(`data:text/javascript,(${work.toString()})()`); - receiver.transform = new RTCRtpScriptTransform(worker, null, null, - {inputCodecs: [customCodec], acceptOnlyInputCodecs = True}); - - function work() { - transform = new TransformStream({ - transform: (frame, controller) => { - // We know that only encrypted-frames will get to this point. - decryptBody(frame); - metadata = frame.metadata(); - metadata.mimeType = 'video/vp8'; - frame.setMetadata(metadata); - controller.enqueue(frame); - } - }); - onrtctransform = async({transformer: {readable, writable, options}}) => - await readable.pipeThrough(transform).pipeTo(writable); - } -}; -``` -# Example code based on the Transform API proposal +# Example code ```js const worker = new Worker(`data:text/javascript,(${work.toString()})()`); @@ -177,7 +91,8 @@ const {codecs} = RTCRtpSender.getCapabilities(); const vp8 = codecs.find(({mimeType}) => mimeType == "video/vp8"); sender.transform = new RTCRtpScriptTransform(worker, { inputCodecs: [vp8], - outputCodecs: [{mimeType: “video/x-encrypted”}] + outputCodecs: [{mimeType: “video/x-encrypted”, + packetizationMode: "video/sframe"}] }); // At receiver side. @@ -220,7 +135,7 @@ function work() { 1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that? - A: Use multiple payload types. Each will be assigned a payload type. Mark each frame with the payload type they need to be packetized as. + A: Use multiple media types. Each will be assigned a payload type. Mark each frame with the media type they need to be packetized as. 1. Q: What is the relationship between this proposal and the IETF standard for SFrame? From 78fa520d94fe357e31ff1ac3cacd172ddceeaa1d Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Tue, 27 Feb 2024 14:35:28 +0100 Subject: [PATCH 39/42] Describe the enabling of codecs for SDP usage --- index.bs | 143 ++++++++++++++++------------------------------- sdp-explainer.md | 6 +- 2 files changed, 50 insertions(+), 99 deletions(-) diff --git a/index.bs b/index.bs index 256d73e..dd3fd50 100644 --- a/index.bs +++ b/index.bs @@ -75,108 +75,21 @@ This specification shows the IDL extensions for WebRTC. It uses an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to insert the processing into the pipeline. -## RTCPeerConnection extensions ## {#rtcpeerconnection-extensions} -In order to correctly identify the transformed content to the other end -of the connection, new interfaces are added on {{RTCPeerConnection}} to -allow the declaration and negotiation of new media types. - -
-// New methods for RTCPeerConnection
-partial interface RTCPeerConnection {
-   undefined addSendCodecCapability(RTCRtpCodecCapability capability);
-   undefined addReceiveCodecCapability(RTCRtpCodecCapability capability);
-};
-
- -In addition, add two new internal slots to the RTCPeerConnection, called -\[[CustomSendCodecs]] -and \[[CustomReceiveCodecs]], -initially empty. - -### Methods ### {#rtcpeerconnection-methods} - -#### AddSendCodecCapability #### {#add-send-codec-capability} - -1. Let 'capability' be the "capability" argument -1. If 'capability' is identical to the capability describing a previously configured codec, - throw an {{InvalidModificationError}} error. -1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot. - -#### AddReceiveCodecCapability #### {#add-receive-codec-capability} - -1. Let 'capability' be the "capability" argument -1. If 'capability' is identical to the capability describing a previously configured codec, - throw an {{InvalidModificationError}} error. -1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot. - -## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions} 
-
 typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
 
-// New methods and attributes for RTCRtpSender and RTCRtpReceiver
+// New methods for RTCRtpSender and RTCRtpReceiver
 partial interface RTCRtpSender {
     attribute RTCRtpTransform? transform;
-    undefined setPacketizer(octet payloadType, RTCRtpCodec packetizer);
 };
 
 partial interface RTCRtpReceiver {
     attribute RTCRtpTransform? transform;
-    undefined setDepacketizer(octet payloadType, RTCRtpCodec depacketizer);
-    undefined addDecodingCodec(octet payloadType, RTCRtpCodec parameters);
 };
-### New methods ### {#sender-receiver-new-methods} -
-
setPacketizer
-
-When called, indicate that the packetizer should behave as if frames -enqueued on the sender with the indicated payload type should be packetized -according to the rules of the codec indicated.
-It is up to the transform to ensure that all data and metadata conforms -to the format required by the packetizer; the sender may drop frames that -do not conform. {{RTCOutboundRtpStreamStats/framesSent}} is not incremented -when a frame is dropped. -
-
setDepacketizer
-
-When called, indicate that incoming packets with the `payloadType` value in -their PT field should be routed to a depacketizer that will perform -depacketization according to the rules of `depacketizer`. -
-
addDecodingCodec
-
-When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument. -
-
- -

-The expected mode of operation is that the sender transform will transform -the frame and change its `payloadType` to the negotiated value for the -transformed content. The receiver transform will receive the `payloadType`, -transform it, and set the `payloadType` to the value that corresponds to the format -of the decoded frame, thus ensuring that it is decoded using the correct -decoder. -

- ## Extension operation ## {#operation} -### Media type negotiation ### {#sdp-negotiation} -The following modifications are applied to the session negotation procedure: - -1. The {{RTCPeerConnection/[[CustomSendCodecs]]}} is included in the list of codecs which the user agent - is currently capable of sending; the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps. - -1. In the [$final steps to create an offer$], step 2.2 is changed to filter the list of codecs against - the union of the [=RTCRtpSender/list of implemented send codecs=] and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the [=list of implemented receive codecs=] and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot. - -1. In the [$final steps to create an answer$], the same modification is done to the filtering in steps 1.2 and 1.3. - - - - -### Codec initialization ### {#codec-initialization} At the time when a codec is initialized as part of the encoder, and the corresponding flag is set in the {{RTCPeerConnection}}'s {{RTCConfiguration}} argument, ensure that the codec is disabled and produces no output. @@ -198,7 +111,7 @@ At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the followin 1. If [=this=].`[[pipeToController]]` is not null, abort these steps. 2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}. - 3. Call pipeTo with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal. + 3. Call pipeTo with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`'s [=AbortController/signal=].

Streams backpressure can optimize throughput while limiting processing and memory consumption by pausing data production as early as possible in a data pipeline. @@ -255,7 +168,7 @@ The `transform` setter steps are: 4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`. 5. Initialize |newPipeToController| to a new {{AbortController}}. 6. If [=this=].`[[pipeToController]]` is not null, run the following steps: - 1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`.signal. + 1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=]. 2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`. 7. Else, run the [$chain transform algorithm$] steps. 8. Set [=this=].`[[pipeToController]]` to |newPipeToController|. @@ -263,13 +176,13 @@ The `transform` setter steps are: 10. Run the steps in the set of [$association steps$] of |transform| with [=this=]. The chain transform algorithm steps are defined as: -1. If |newPipeToController|.signal is [=AbortSignal/aborted=], abort these steps. +1. If |newPipeToController|'s [=AbortController/signal=] is [=AbortSignal/aborted=], abort these steps. 2. [=ReadableStreamDefaultReader/Release=] |reader|. 3. [=WritableStreamDefaultWriter/Release=] |writer|. 4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`. -5. Call pipeTo with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal. -6. Call pipeTo with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal. +5. Call pipeTo with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|'s [=AbortController/signal=]. +6. Call pipeTo with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|'s [=AbortController/signal=]. This algorithm is defined so that transforms can be updated dynamically. There is no guarantee on which frame will happen the switch from the previous transform to the new transform. @@ -448,6 +361,25 @@ dictionary RTCEncodedVideoFrameMetadata { ### Members ### {#RTCEncodedVideoFrameMetadata-members}

+
+ frameId unsigned long long +
+
+

+ An identifier for the encoded frame, monotonically increasing in decode order. Its lower + 16 bits match the frame_number of the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]], if present. + Only present for received frames if the Dependency Descriptor Header Extension is present. +

+
+
+ dependencies sequence<unsigned long long> +
+
+

+ List of frameIds of frames this frame references. + Only present for received frames if the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]] is present. +

+
synchronizationSource unsigned long
@@ -859,6 +791,13 @@ interface KeyFrameRequestEvent : Event { constructor(DOMString type, optional DOMString rid); readonly attribute DOMString? rid; }; + +dictionary RTCRtpScriptTransformParameters { + sequence<RTCRtpCodecParameters> inputCodecs; + sequence<RTCRtpCodecParameters> outputCodecs; + boolean acceptOnlyInputCodecs = false; +}; + ## Operations ## {#RTCRtpScriptTransform-operations} @@ -868,6 +807,7 @@ The generateKeyFrame(|rid|) method steps are: 1. Let |promise| be a new promise. 1. Run the [$generate key frame algorithm$] with |promise|, |this|.`[[encoder]]` and |rid|. diff --git a/sdp-explainer.md b/sdp-explainer.md index a4fb3e4..1c5c206 100644 --- a/sdp-explainer.md +++ b/sdp-explainer.md @@ -43,8 +43,8 @@ We extend the RTCRTPScriptTransform object's constructor with a fourth argument ``` dictionary CodecInformation { - sequence inputCodecs; - sequence outputCodecs; + sequence<RTCRtpCodecCapabilityWithPacketization> inputCodecs; + sequence<RTCRtpCodecCapabilityWithPacketization> outputCodecs; bool acceptOnlyInputCodecs = false; } ``` @@ -59,8 +59,6 @@ negotiate in the SDP, and it serves to filter the frame types that the transform In order to be able to use the filtering function, the "acceptOnlyInputCodecs" has to be set to true; if it is false, all frames are delivered to the transform. -The next section has two versions - the Transceiver proposal and the Transform proposal. - ## For SDP negotiation When the PeerConnection generates an offer or an answer: From d9b390d0c0bc6d400fd963bf9b938de9639a24ca Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Wed, 17 Apr 2024 15:38:57 +0200 Subject: [PATCH 40/42] Fix acceptOnlyInputCodecs --- index.bs | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/index.bs b/index.bs index dd3fd50..cb3a1cb 100644 --- a/index.bs +++ b/index.bs @@ -130,6 +130,7 @@ The readEncodedData algorithm is given a |rtcObject| as p 1. Let |frame| be the newly produced frame. 1. Set |frame|.`[[owner]]` to |rtcObject|. 1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`. +1. If |rtcObject|'s |transform|'s |options| contains {{RTCRtpScriptTransformParameters/acceptOnlyInputCodecs}} with the value True, and the {{RTCEncodedVideoFrameMetadata/mimeType}} of |frame| does not match any codec in |transform|'s |options|' |inputCodecs|, execute the [$writeEncodedData$] algorithm given |rtcObject| and |frame|. 1. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`. The writeEncodedData algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps: @@ -146,7 +147,7 @@ The writeEncodedData algorithm is given a |rtcObject| as On sender side, as part of [$readEncodedData$], frames produced by |rtcObject|'s encoder MUST be enqueued in |rtcObject|.`[[readable]]` in the encoder's output order. As [$writeEncodedData$] ensures that the transform cannot reorder frames, the encoder's output order is also the order followed by packetizers to generate RTP packets and assign RTP packet sequence numbers. -The packetizer may expect the transformed data to still conform to the original format, e.g. a series of NAL units separated by Annex B start codes. +The packetizer will expect the transformed data to conform to the format indicated by its {{RTCEncodedVideoFrameMetadata/mimeType}}. On receiver side, as part of [$readEncodedData$], frames produced by |rtcObject|'s packetizer MUST be enqueued in |rtcObject|.`[[readable]]` in the same encoder's output order. To ensure the order is respected, the depacketizer will typically use RTP packet sequence numbers to reorder RTP packets as needed before enqueuing frames in |rtcObject|.`[[readable]]`. @@ -841,10 +842,6 @@ Each RTCRtpScriptTransform has the following set of [$association steps$], given 1. Set |transformer|.`[[encoder]]` to |encoder|. 1. Set |transformer|.`[[depacketizer]]` to |depacketizer|. -// FIXME: Describe the influence of the acceptOnlyInputCodecs on frame routing. - -// FIXME: Describe the influence of the "packetizer" argument of a codec on encoder and depacketizer. - The generateKeyFrame(|rid|) method steps are: 1. Let |promise| be a new promise. 1. Run the [$generate key frame algorithm$] with |promise|, |this|.`[[encoder]]` and |rid|. From 604da655fea455f6c6632fbbd66aa9b59a874b9c Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Thu, 18 Apr 2024 10:10:30 +0200 Subject: [PATCH 41/42] Added packetizationMode description --- index.bs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/index.bs b/index.bs index cb3a1cb..b7c530e 100644 --- a/index.bs +++ b/index.bs @@ -801,6 +801,17 @@ dictionary RTCRtpScriptTransformParameters { +### Extension to RTCRtpCodec ### {#RTCRtpCodec-extensions} + +The member "packetizationMode" is added to the {{RTCRtpCodec}} definition; its value is a MIME type. When this is present, the rules for packetizing and depacketizing are those of the named MIME type. For codecs not supported by the platform, this MUST be present. + +
+partial dictionary RTCRtpCodec {
+  DOMString packetizationMode;
+};
+
+ + ## Operations ## {#RTCRtpScriptTransform-operations} The new RTCRtpScriptTransform(|worker|, |options|, |transfer|) constructor steps are: From 36fd11b924269a8dee58eaae9db5ab70576c79cc Mon Sep 17 00:00:00 2001 From: Harald Alvestrand Date: Mon, 22 Apr 2024 11:41:45 +0200 Subject: [PATCH 42/42] Use Codec not CodecParameters --- index.bs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/index.bs b/index.bs index b7c530e..8b109fb 100644 --- a/index.bs +++ b/index.bs @@ -794,8 +794,8 @@ interface KeyFrameRequestEvent : Event { }; dictionary RTCRtpScriptTransformParameters { - sequence<RTCRtpCodecParameters> inputCodecs; - sequence<RTCRtpCodecParameters> outputCodecs; + sequence<RTCRtpCodec> inputCodecs; + sequence<RTCRtpCodec> outputCodecs; boolean acceptOnlyInputCodecs = false; };