Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add description of an API for controlling SDP codec negotiation #186

Open
wants to merge 43 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
a14a855
Create sdp-explainer.md
alvestrand May 30, 2023
005121c
Fix typos
Orphis May 30, 2023
8c5ec20
Add decryptedPT, erroneous frame handling
alvestrand Jun 2, 2023
9c7ba95
Add specification
alvestrand Jun 13, 2023
6c6ca27
Fix internal slots (or die trying)
eladalon1983 Jun 14, 2023
d3c6ec5
Fix linker
eladalon1983 Jun 14, 2023
1342689
More sensible errors to throw
alvestrand Jun 14, 2023
bf6e469
Lowercase functions, add linkages
alvestrand Jun 15, 2023
9d8212c
Update sdp-explainer.md
alvestrand Jun 20, 2023
cba1936
Update sdp-explainer.md
alvestrand Jun 20, 2023
871a6bb
Update sdp-explainer.md
alvestrand Jun 20, 2023
2cd3e39
Update sdp-explainer.md
alvestrand Jun 20, 2023
ebdb6fc
Some review comments addressed on index.bs
alvestrand Aug 29, 2023
c0bf964
Updated sdp-explainer.md with new code++
alvestrand Aug 29, 2023
ef896ab
Introduce setPacketizer
alvestrand Oct 6, 2023
d582ea5
Minifix
alvestrand Oct 6, 2023
a35b9fd
Minifix
alvestrand Oct 6, 2023
d433822
Apply suggestions from code review
alvestrand Oct 8, 2023
713a771
Fix a couple of other references
dontcallmedom Oct 9, 2023
5f483bc
Explainer update
alvestrand Oct 13, 2023
deb9866
Fix md formatting
alvestrand Oct 13, 2023
a02ac43
Explain in terms of transceiver
alvestrand Nov 13, 2023
5cecfe3
Sframe -> SFrame
alvestrand Nov 14, 2023
f3be9dd
Update sdp-explainer.md
alvestrand Nov 16, 2023
abd0235
Update sdp-explainer.md
alvestrand Nov 17, 2023
4e34b4c
Update sdp-explainer.md
alvestrand Nov 17, 2023
97f16ac
Update sdp-explainer.md
alvestrand Nov 17, 2023
d779d39
Fix example code
alvestrand Nov 21, 2023
8abaef4
Merge branch 'w3c:main' into sdp-negotiation
alvestrand Dec 15, 2023
b44d67f
Add "both alternatives" API description
alvestrand Dec 15, 2023
2958844
Update sdp-explainer.md
alvestrand Jan 12, 2024
78fbfbc
Update sdp-explainer.md
alvestrand Jan 12, 2024
a975401
Update sdp-explainer.md
alvestrand Jan 25, 2024
8aa1533
Update index.bs
alvestrand Jan 25, 2024
bd0c908
Update index.bs
alvestrand Jan 25, 2024
bbc7f6b
Update index.bs
alvestrand Jan 25, 2024
0ca754b
Update sdp-explainer.md
alvestrand Jan 25, 2024
ccf6ea8
Fix "pre" section end
alvestrand Jan 25, 2024
6206a46
Update sdp explainer to remove transceiver alternative
alvestrand Feb 27, 2024
78fa520
Describe the enabling of codecs for SDP usage
alvestrand Feb 27, 2024
d9b390d
Fix acceptOnlyInputCodecs
alvestrand Apr 17, 2024
604da65
Added packetizationMode description
alvestrand Apr 18, 2024
36fd11b
Use Codec not CodecParameters
alvestrand Apr 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 88 additions & 1 deletion index.bs
Original file line number Diff line number Diff line change
Expand Up @@ -61,21 +61,108 @@ This specification shows the IDL extensions for WebRTC.
It uses an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to
insert the processing into the pipeline.

## RTCPeerConnection extensions ## {#rtcpeerconnection-extensions}
In order to correctly identify the transformed content to the other end
of the connection, new interfaces are added on {{RTCPeerConnection}} to
allow the declaration and negotiation of new media types.

<pre class="idl">
// New methods for RTCPeerConnection
partial interface RTCPeerConnection {
undefined AddSendCodecCapability(RTCRtpCodecCapability capability);
undefined AddReceiveCodecCapability(RTCRtpCodecCapability capability);
alvestrand marked this conversation as resolved.
Show resolved Hide resolved
};
</pre>

In addition, add two new internal slots to the RTCPeerConnection, called
<dfn attribute dfn-for="RTCPeerConnection">\[[CustomSendCodecs]]</dfn>
and <dfn attribute dfn-for="RTCPeerConnection">\[[CustomReceiveCodecs]]</dfn>,
initially empty.

### Methods ### {#rtcpeerconnection-methods}

#### AddSendCodecCapability #### {#add-send-codec-capability}

1. Let 'capability' be the "capability" argument
1. If 'capability' is identical to the capability describing a previously configured codec,
throw an IllegalModification error.
alvestrand marked this conversation as resolved.
Show resolved Hide resolved
1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomSendCodecs]]}} internal slot.

#### AddReceiveCodecCapability #### {#add-receive-codec-capability}

1. Let 'capability' be the "capability" argument
1. If 'capability' is identical to the capability describing a previously configured codec,
throw an IllegalModification error.
alvestrand marked this conversation as resolved.
Show resolved Hide resolved
1. Add 'capability' to the PeerConnections's {{RTCPeerConnection/[[CustomReceiveCodecs]]}} internal slot.

## Extensions to RTCRtpSender and RTCRtpReceiver ## {#sender-receiver-extensions}
<pre class="idl">

typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;

// New methods for RTCRtpSender and RTCRtpReceiver
// New methods and attributes for RTCRtpSender and RTCRtpReceiver
partial interface RTCRtpSender {
attribute RTCRtpTransform? transform;
undefined setPacketizer(octet payloadType, RTCRtpCodec packetizer);
};

partial interface RTCRtpReceiver {
attribute RTCRtpTransform? transform;
undefined setDepacketizer(octet payloadType, RTCRtpCodec depacketizer);
undefined addDecodingCodec(octet payloadType, RTCRtpCodec parameters);
};
</pre>

### New methods ### {#sender-receiver-new-methods}
<dl>
<dt>setPacketizer</dt>
<dd>
When called, indicate that the packetizer should behave as if frames
enqueued on the sender with the indicated payload type should be packetized
according to the rules of the codec indicated.<br>
It is up to the transform to ensure that all data and metadata conforms
to the format required by the packetizer; the sender may drop frames that
do not conform. {{RTCOutboundRtpStreamStats/framesSent}} is not incremented
when a frame is dropped.
</dd>
<dt>setDepacketizer</dt>
<dd>
When called, indicate that incoming packets with the `payloadType` value in
their PT field should be routed to a depacketizer that will perform
depacketization according to the rules of `depacketizer`.
</dd>
<dt>addDecodingCodec</dt>
<dd>
When called, initialize a decoder in the RTCRtpReceiver to handle the format indicated by the `parameters` argument.
</dd>
</dl>

<p class="note">
The expected mode of operation is that the sender transform will transform
the frame and change its `payloadType` to the negotiated value for the
transformed content. The receiver transform will receive the `payloadType`,
transform it, and set the `payloadType` to the value that corresponds to the format
of the decoded frame, thus ensuring that it is decoded using the correct
decoder.
</p>

## Extension operation ## {#operation}

### Media type negotiation ### {#sdp-negotiation}
The following modifications are applied to the session negotation procedure:

1. The {{RTCPeerConnection/[[CustomSendCodecs]]}} is included in the list of codecs which the user agent
is currently capable of sending; the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} is included in the list of codecs which the user agent is currently prepared to receive, as referenced in "set a session description" steps.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again I wonder if these custom codecs could instead just live in transceiver.[[PreferredCodecs]]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 8 in setCodecPreferences() filters against codecCapabilities (the static).
If we modify that, it would be possible - but would we entirely stop filtering, then?


1. In the [$final steps to create an offer$], step 2.2 is changed to filter the list of codecs against
the union of the [=RTCRtpSender/list of implemented send codecs=] and the {{RTCPeerConnection/[[CustomSendCodecs]]}} slot, and step 2.3 is changed to filter against the union of the [=list of implemented receive codecs=] and the {{RTCPeerConnection/[[CustomReceiveCodecs]]}} slot.

1. In the [$final steps to create an answer$], the same modification is done to the filtering in steps 1.2 and 1.3.




### Codec initialization ### {#codec-initialization}
At the time when a codec is initialized as part of the encoder, and the
corresponding flag is set in the {{RTCPeerConnection}}'s {{RTCConfiguration}}
argument, ensure that the codec is disabled and produces no output.
Expand Down
137 changes: 137 additions & 0 deletions sdp-explainer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# SDP negotiation in Encoded Transform

The original definition of encoded transform did not consider negotation; the frames on the "transformed" side went out stamped with the payload type of the frame that came in on the "non-transformed" side (and vice versa for the receiver).
alvestrand marked this conversation as resolved.
Show resolved Hide resolved

This creates a problem, in that when an encoded transform is applied on the sending side, the bits on the wire may not correspond to what the SDP negotiation has declared that particular type to be used for. When only talking to another instance of the same application, that is not an issue, but it is an issue as soon as we want two different applications to interoperate - for instance when exchanging SFrame encrypted media between two endpoints from different application providers, or when exchanging SFrame encrypted content via an SFU that expects to be able to decode or modify the media.

(The latter is exactly what Sframe is designed to prevent, but it is better for the intermediary to fail clean than to engage in possibly random behavior due to attempting to decode a stream that does not conform to the description it expects.)
alvestrand marked this conversation as resolved.
Show resolved Hide resolved

This problem is even more acute when the Encoded Transform is used to add support for codecs not natively supported by the browser; without the ability to influence SDP negotiation, there is no standard way to ensure that a receiver supporting the new codec is able to associate the payload type of incoming packets with the right decoder.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "codecs in JS" use case lacks consensus from last meeting and should be removed.


For example, it's been proposed to add [Lyra](https://github.com/google/lyra) to WebRTC using an implementation in WASM
- a working example using SDP munging can be found on the
[Meetecho blog](https://www.meetecho.com/blog/playing-with-lyra/).

# Requirements for an SDP negotiation API
The following things need to be available on such an API:
1. Before SDP negotiation, the application must be able to specify one or more new media types that one wants to negotiate for. As a point of illustration, this document uses the type "video/new-codec".
2. After SDP negotiation, the application must be able to identify if negotiation has succeeded or failed, and what payload type has been assigned for the new media type.
3. Before sending frames, the application must be able to inform the RTP sender of what kind of packetization to use on the outgoing frames.
4. Before receiving frames, the application must be able to inform the RTP receiver of what kind of depacketization to use on the incoming frames.
alvestrand marked this conversation as resolved.
Show resolved Hide resolved
5. When transforming frames, the sending application must be able to mark the transformed frames with the negotiated payload type before sending.
6. When transforming frames, the receiving application must be able to check that the incoming frame has the negotiated payload type, and (if reenqueueing the frame after transformation) mark the transformed frame with the appropriate payload type for decoding within the RTPReceiver.

# API description

## Codec description
For codec description, we reuse the dictionary RTCRtpCodecCapability.
alvestrand marked this conversation as resolved.
Show resolved Hide resolved

The requirements on the parameters are:
- either mimetype or fmtp parameters must be different from any existing capability
alvestrand marked this conversation as resolved.
Show resolved Hide resolved

When a codec capability is added, the SDP machinery will negotiate these codecs as normal, and the resulting payload type will be visible in RTCRtp{Sender,Receiver}.getParameters().


## For SDP negotiation
SDP negotiation is inherently an SDP property, and the proper target for that is therefore the RTCRtpTransceiver object.

The transceiver will always be available before an offer or answer is created (either because of AddTrack or AddTransceiver on the sender side,
or in the ontrack event on the side that receives the offer). At that time, the additional codecs to negotiate must be set:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ontrack is there a design issue here? If you want to add a codec, you can only do that after SRD which triggers ontrack. I know an implementation that will probably not like this.

This is also where I think developers will stumble over the per-transceiver approach. At least RtpCodec does not specify the payload type or we might add a footgun for breaking RTP demuxing rules.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you add a codec that is not understood to the SDP it should not show up in getParameters().codecs.
But it seems we have a problem since receiver.getParameters().codecs is empty before negotiation? (might be a bug, this rings a bell...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"not understood to the SDP"...?
Yes, that is why RtpCodec is the right interface. The PT is a product of the negotiation, not input to it.

Once ontrack fires, we're in the middle of negotiation, is receiver.getParameters() .codecs filled out at that time?

Copy link
Collaborator

@fippo fippo Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you add a codec not understood by the receiver to the SDP ...

is receiver.getParameters() .codecs filled out at that time

No they are not which might be an instance of https://bugs.chromium.org/p/webrtc/issues/detail?id=12116


```
transceiver.addCodec(RTCRtpCodec codec, RTCRtpCodec packetizationMode)
```
This will add the described codec to the list of codecs to be offered in createOffer/createAnswer. On successful negotiation
of the codec, it will appear in the negotiated codec list returned from setParameters().

Furthermore, the call will instruct the sender to packetize, and the receiver to depacketize, in accordance with the rules for the
codec being given as the packetizationMode argument.

Because it is sometimes inconvenient to intercept every call that creates transceivers, a convenience method is offered on the PC level:

```
pc.addCodecCapability(DOMString kind, RTCRtpCodec codec, RTCRtpCodec packetizationMode)
```
These calls will act as if the addCodec() call had been invoked on every transceiver created of the associated "kind".

NOTE: The codecs will not show up on the static sender/receiver getCapabilities methods, since these methods can’t distinguish between capabilities used for different PeerConnections. They will show up in the list of codecs in RTCRtp{Sender,Receiver}.getParameters(), so they’re available to the RTCRtpSender for selection or deselection.



## Existing APIs that will be used together with the new APIs
- Basic establishing of EncodedTransform
- getParameters() to get results of codec negotiation
- encoded frame SetMetadata, to set the payload type for processed frames

# Example code
(This is incomplete)
```
customCodec = {
mimeType: “video/x-encrypted”,
clockRate: 90000,
fmtp = “encapsulated-codec=vp8”,
};

// At sender side
pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"});
alvestrand marked this conversation as resolved.
Show resolved Hide resolved

// ...after negotiation

const {codecs} = sender.getParameters();
const {payloadType} = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted");

const worker = new Worker(`data:text/javascript,(${work.toString()})()`);
sender.transform = new RTCRtpScriptTransform(worker, {payloadType});

alvestrand marked this conversation as resolved.
Show resolved Hide resolved
function work() {
onrtctransform = async ({transformer: {readable, writable, options}}) =>
await readable.pipeThrough(new TransformStream({transform})).pipeTo(writable);

function transform(frame, controller) {
// transform chunk
let metadata = frame.metadata();
encryptBody(frame);
metadata = frame.metadata();
metadata.pt = options.payloadType;
frame.setMetadata(metadata);
controller.enqueue(frame);
}
}

// At receiver side.
const decryptedPT = 208; // Can be negotiated PT or locally-valid
pc.addCodecCapability('video', customCodec, {mimeType: "video/vp8"});
alvestrand marked this conversation as resolved.
Show resolved Hide resolved
pc.ontrack = ({receiver}) => {
const {codecs} = sender.getParameters();
const encryptingCodec = codecs.find(({mimeType}) => mimeType == "video/acme-encrypted");
receiver.setDepacketizer(encryptingCodec.payloadType, {mimeType: "video/vp8"});
alvestrand marked this conversation as resolved.
Show resolved Hide resolved
receiver.addDecodingCodec({mimeType: 'video/vp8', payloadType: decryptedPT});
const worker = new Worker(`data:text/javascript,(${work.toString()})()`);
receiver.transform = new RTCRtpScriptTransform(worker, {payloadType});

function work() {
transform = new TransformStream(
transform: (frame, controller) => {
if (frame.metadata().payloadType != encryptedPT) {
continue; // Ignore all frames for the wrong payload.
}
decryptBody(frame);
metadata = frame.metadata();
metadata.payloadType = decryptedPT;
controller.enqueue(frame);
}
});
onrtctransform = async({transformer: {readable, writable, options}}) =>
await readable.pipeThrough(transform).pipeTo(writable);
};
```

alvestrand marked this conversation as resolved.
Show resolved Hide resolved
# Frequently asked questions

1. Q: My application wants to send frames with multiple packetizers. How do I accomplish that?

A: Use multiple payload types. Each will be assigned a payload type. Mark each frame with the payload type they need to be packetized as.

1. Q: What is the relationship between this proposal and the IETF standard for SFrame?

A: This proposal is intended to make it possible to implement the IETF standard for SFrame in Javascript, with only the packetizer/depacketizer being updated to support the SFrame packetization rules. It is also intended to make it possible to perform other forms of transform, including the ones that are presently deployed in the field, while marking the SDP with a truthful statement of its content.
Loading