Add RTCRtpEncodingParameters.codec to change the active codec #147

Orphis · 2023-03-16T14:58:07Z

This new encoding parameter allows changing the active codec on any simulcast layer as long as it is supported or negotiated.

It allows user agent that don't support the feature partially or fully to reject the configuration change.

Fixes #43 , #126

Preview | Diff

index.html

henbos · 2023-03-16T16:14:43Z

index.html

+          different codec values on the different encodings, return a promise [= rejected =]
+          with a newly [= exception/created =] {{OperationError}}.
+        </p>
+      </li>


In parallel, reconfigure the encoder? Edit: maybe superflous, we already have language "In parallel, configure the media stack to use parameters to transmit sender.[[SenderTrack]]."

I think the UA should be able to determine ahead of time if it is able to apply this type of configuration, so it should be able to reject it early, without a parallel step. setParameters() does have a provision for handling other errors with an OperationError which would cover late discovery too.

Though addTransceiver() does not have such error handling. What are we supposed to do if we run out of HW resources when using it?

Luckily addTransceiver() will not acquire any resources, because it hasn't been negotiated yet. The spec currently says to acquire during negotiation, but as pointed out by Bernard in the last Virtual interim meeting, in practise we don't acquire encoder resources until when we actually use them, so any failure would have to be surfaced then - this is an existing problem, and perhaps we'll want to event handlers for this in the future, but that's a separate issue

I think the UA should be able to determine ahead of time if it is able to apply this type of configuration, so it should be able to reject it early, without a parallel step.

Depends on your definition of "apply". UAs may be able to reject some combinations early, but it's not clear to me that success is guaranteed any more strongly for customized codecs than it is for default ones.

In any case, while rejecting a promise early has some benefit while stepping through in a JS debugger, there's not much else to gain from that particular timing.

henbos · 2023-03-16T16:26:11Z

Looks great, it just needs some clarifications

Orphis · 2023-03-21T14:49:27Z

Note: the current version still lacks fallback behaviour on negotiation (the requested codec is negotiated away and the codec field should be cleared).

Orphis · 2023-03-21T15:01:30Z

@dontcallmedom Could you help me figure out why {{RTCRtpTransceiver/[[PreferredCodecs]]}} is not working?

dontcallmedom · 2023-03-21T15:05:20Z

the problem is that the definition of {{RTCRtpTransceiver/[[PreferredCodecs]]}} in webrtc-pc is marked as not exported - I'll look into fixing it

needed for w3c/webrtc-extensions#147 (comment)

henbos

LGTM % one minor clarification

index.html

jan-ivar · 2023-03-27T21:17:34Z

index.html

@@ -555,7 +556,102 @@ <h2>Dictionary {{RTCRtpEncodingParameters}} Members</h2>
        allows the <a>user agent</a> to adapt its bandwidth allocation strategy
        based on the current network conditions.</p>
      </dd>
+      <dt><dfn data-idl>codec</dfn> of type <span class="idlMemberType">RTCRtpCodec</span></dt>
+      <dd>
+        <p>Optional value overriding the codec used for this encoding. The {{RTCRtpCodec}} dictionary is


The slides presented justified the ask for simulcast, which is video only. So I think we have agreement for this API for video only.

Suggested change

<p>Optional value overriding the codec used for this encoding. The {{RTCRtpCodec}} dictionary is

<p>This member can only be present if the sender's <code class=

"gum">kind</code> is `"video"`.

Optional value overriding the codec used for this encoding. The {{RTCRtpCodec}} dictionary is

We should make sure this rejects for audio if present on input, like we do for the other video-specific things.

The "negotiation once, never re-negotiate" use case is becoming more popular, where one uses CSRCs to identify which participant is which, but always having a fixed number of transceivers.

When you can do negotiationless codec switching for video, I think it is only a matter of time before there will be an ask for audio as well.

That said the audio use case seems a lot more "at risk" than the video use case. For now I don't mind limiting scope to video-only, but I don't think we should rule out the possibility long term.

One of the bullet points on this slide is "make it possible to change codec without re-negotiating". But this API is much more important for video than for audio due to the other two bullet points which are both video-only uses:

Allow different codecs on different encodings.

Allow specifying both codec and scalabilityMode with a single API call.

I don't think we should restrict this API to video only. Codec is a general concept, why should it only apply for video only?
While the slides were presented with examples centered around video, with the emergence of new audio codecs, it seems even more important to keep the symmetry in the APIs.

The slides presented justified the ask for simulcast, which is video only. So I think we have agreement for this API for video only.

Simulcast as a concept is becoming important for audio too, e.g. for cases that want to do PSTN gateway for some participants without transcoding which might mean sending Opus and G722 (or G711) in the same packet.

This API has video use-cases without simulcast too, e.g. you might want to switch codecs because the far end now needs you to downgrade from a modern codec to the video-equivalent of G711 for reasons.

I do not see a reason to artificially restrict the specification.

index.html

jan-ivar · 2023-03-27T22:21:35Z

index.html

+          different codec values on the different encodings, return a promise [= rejected =]
+          with a newly [= exception/created =] {{OperationError}}.
+        </p>
+      </li>


I think the UA should be able to determine ahead of time if it is able to apply this type of configuration, so it should be able to reject it early, without a parallel step.

Depends on your definition of "apply". UAs may be able to reject some combinations early, but it's not clear to me that success is guaranteed any more strongly for customized codecs than it is for default ones.

In any case, while rejecting a promise early has some benefit while stepping through in a JS debugger, there's not much else to gain from that particular timing.

index.html

jan-ivar · 2023-03-27T22:24:05Z

index.html

+        <p>If the user agent does not support changing the codec for any encoding or mixing
+          different codec values on the different encodings, 


"mixing different codec values" seems a bit vague. What exactly are the requirements for not throwing OperationError?

In practice, does this just mean "doesn't support the codec attribute"? Do existing browsers throw an OperationError?

OperationError is already part of the specification when setParameters() cannot allocate resources. Chrome is able to throw those when unknown errors occur deep in the pipeline, which we associate with a resource allocation issue for now.
This describes a new behaviour though and I believe returning an OperationError in case of a complex unsupported input might be relevant.

dontcallmedom-bot · 2023-03-29T07:51:14Z

This issue was discussed in the minutes of WebRTC March 2023 meeting – 21 March 2023 (PR #147 Add RTCRtpEncodingParameters.codec to change the active codec)

aboba

The checks for setParameters() seem to allow for setting of combinations of codec and scalabilityMode that are illegal (e.g. setting VP8 and L3T3). To fix this, I think we need to add a check in WebRTC-SVC.

jan-ivar · 2023-04-05T20:12:15Z

index.html

+      <p>Add the following steps in the parameter validation for {{RTCRtpSender/setParameters()}},
+        before any validation of <var>scalabilityMode</var> from [[WEBRTC-SVC]]:</p>


This adds an unnecessary normative reference to [[WEBRTC_SVC]], which we can avoid. I also struggled with where exactly it meant, and readers shouldn't have to look up a third spec to find out. scalebilityMode also seems irrelevant. Can we express it differently? Same for addTransceiver below.

Looking up https://w3c.github.io/webrtc-svc/#setparameters it says: "[WEBRTC] Section 5.2 describes validation of parameters within setParameters(). Add the following to the conditions under which the operation causes a promise rejected with an InvalidModificationError (step 4 within step 6):"

That doesn't seem to be the right place. Where exactly is this to be inserted?

I've added w3c/webrtc-pc#2854 to try to help.

The problem is that not that the normative reference is incorrect - the problem is that the combined procedure doesn't produce the results you'd expect.

My goal there is to make sure that the scalabilityMode value is checked against the capabilities associated with codec and not against the general capabilities, codec preferences or first codec in the negotiated list as mentioned in WEBRTC-SVC. Any guidance to do this correctly would be appreciated.

jan-ivar · 2023-04-05T21:26:45Z

index.html

+      <p>Add the following steps in the parameter validation for {{RTCRtpSender/setParameters()}},
+        before any validation of <var>scalabilityMode</var> from [[WEBRTC-SVC]]:</p>
+      <ol>
+        <li><p>Let <var>codecs</var> be <var> parameters</var>.{{RTCRtpParameters/codecs}}.</p>


Hmm, webrtc-pc already contains this (unused!) line, so assuming w3c/webrtc-pc#2854 gets merged, how about something like: "Under the [=setParameters validation steps=]", replace this line:

Let codecs be parameters.codecs.

with:

Let choosableCodecs be parameters.codecs.

If choosableCodecs is an empty list, set choosableCodecs to transceiver.{{RTCRtpTransceiver/[[PreferredCodecs]]}}

If choosableCodecs is still an empty list, set choosableCodecs to the [=list of implemented send codecs=] for [=transceiver kind=]

And then follow SVC's lead above and say: "Add the following to the conditions under which the operation causes a promise rejected with an InvalidModificationError (step 4 within step 6):"

Any encoding in encodings [=map/exists|contains=] a codec [= codec match | not found =] in choosableCodecs

This should avoid the if/else/else and hopefully be readable. WDYT?

To precent attempts to set illegal combinations of codecs and scalabilityMode values, I think you need to also set the "most preferred codec" slot and refer to this slot in WebRTC-SVC Section 4.2.2.

aboba

I don't think that the proposed checks work together with the WebRTC-SVC checks to produce the expected results. See other comments.

jan-ivar

whoops, had this pending comment from before that was not submitted

index.html

dontcallmedom-bot · 2023-04-19T14:26:40Z

This issue had an associated resolution in WebRTC WG 2023-04-18 – (PR 147: Add RTCRtpEncodingParameters.codec to change the active codec (Florent)):

RESOLUTION: No objection, let’s include audio.

This new encoding parameter allows changing the active codec on any simulcast layer as long as it is supported or negotiated. It allows user agent that don't support the feature partially or fully to reject the configuration change. Fixes w3c#43 , w3c#126

index.html

Co-authored-by: Jan-Ivar Bruaroey <jan-ivar@users.noreply.github.com>

dontcallmedom-bot · 2023-06-28T06:35:04Z

This issue was mentioned in WEBRTCWG-2023-06-27 (Page 46)

Orphis requested review from aboba and jan-ivar March 16, 2023 14:58

dontcallmedom linked an issue Mar 16, 2023 that may be closed by this pull request

Negotiationless codec selection and per-encoding layer codec configuration #126

Closed

henbos reviewed Mar 16, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

henbos reviewed Mar 16, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

henbos reviewed Mar 16, 2023

View reviewed changes

Orphis force-pushed the add_codec branch from a1e6ac2 to 13e5eb9 Compare March 21, 2023 14:42

dontcallmedom added a commit to w3c/webrtc-pc that referenced this pull request Mar 21, 2023

Mark PreferredCodec internal slot as exported

625bef4

needed for w3c/webrtc-extensions#147 (comment)

dontcallmedom mentioned this pull request Mar 21, 2023

Mark PreferredCodec internal slot as exported w3c/webrtc-pc#2843

Merged

Orphis force-pushed the add_codec branch from 13e5eb9 to f09e06a Compare March 23, 2023 13:29

henbos approved these changes Mar 23, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

youennf reviewed Mar 23, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

jan-ivar reviewed Mar 27, 2023

View reviewed changes

w3c deleted a comment from dontcallmedom-bot Mar 29, 2023

aboba reviewed Mar 30, 2023

View reviewed changes

aboba mentioned this pull request Mar 30, 2023

clean up audio rtp parameters extensions #154

Merged

Orphis force-pushed the add_codec branch from f09e06a to deace94 Compare March 30, 2023 14:55

henbos approved these changes Mar 30, 2023

View reviewed changes

jan-ivar mentioned this pull request Apr 5, 2023

Export links to addTransceiver/setParameters steps + unnumber some steps w3c/webrtc-pc#2854

Merged

jan-ivar reviewed Apr 5, 2023

View reviewed changes

aboba requested changes Apr 6, 2023

View reviewed changes

jan-ivar reviewed Apr 11, 2023

View reviewed changes

index.html Show resolved Hide resolved

Orphis force-pushed the add_codec branch from deace94 to 6bc5eec Compare April 20, 2023 13:41

Orphis requested review from aboba and jan-ivar April 20, 2023 13:43

aboba approved these changes Apr 20, 2023

View reviewed changes

jan-ivar reviewed Apr 20, 2023

View reviewed changes

index.html Outdated Show resolved Hide resolved

Update index.html

a088a4f

Co-authored-by: Jan-Ivar Bruaroey <jan-ivar@users.noreply.github.com>

jan-ivar approved these changes Apr 20, 2023

View reviewed changes

aboba merged commit 1ae522b into w3c:main Apr 20, 2023
2 checks passed

Orphis mentioned this pull request Apr 24, 2023

WebRTC Codec selection API w3ctag/design-reviews#836

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add RTCRtpEncodingParameters.codec to change the active codec #147

Add RTCRtpEncodingParameters.codec to change the active codec #147

Orphis commented Mar 16, 2023 •

edited by pr-preview bot

henbos Mar 16, 2023 •

edited

Orphis Mar 21, 2023

henbos Mar 23, 2023

jan-ivar Mar 27, 2023

henbos commented Mar 16, 2023

Orphis commented Mar 21, 2023

Orphis commented Mar 21, 2023

dontcallmedom commented Mar 21, 2023

henbos left a comment

jan-ivar Mar 27, 2023

henbos Mar 28, 2023

henbos Mar 28, 2023

Orphis Mar 30, 2023

fippo Mar 31, 2023 •

edited

jan-ivar Mar 27, 2023

jan-ivar Mar 27, 2023

aboba Mar 30, 2023

Orphis Mar 30, 2023

dontcallmedom-bot commented Mar 29, 2023 •

edited by dontcallmedom

aboba left a comment •

edited

jan-ivar Apr 5, 2023

aboba Apr 6, 2023

Orphis Apr 6, 2023

jan-ivar Apr 5, 2023

aboba Apr 6, 2023

aboba left a comment

jan-ivar left a comment

dontcallmedom-bot commented Apr 19, 2023

dontcallmedom-bot commented Jun 28, 2023

		<p>If the user agent does not support changing the codec for any encoding or mixing
		different codec values on the different encodings,

		<p>Add the following steps in the parameter validation for {{RTCRtpSender/setParameters()}},
		before any validation of <var>scalabilityMode</var> from [[WEBRTC-SVC]]:</p>

Add RTCRtpEncodingParameters.codec to change the active codec #147

Add RTCRtpEncodingParameters.codec to change the active codec #147

Conversation

Orphis commented Mar 16, 2023 • edited by pr-preview bot

henbos Mar 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henbos commented Mar 16, 2023

Orphis commented Mar 21, 2023

Orphis commented Mar 21, 2023

dontcallmedom commented Mar 21, 2023

henbos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fippo Mar 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dontcallmedom-bot commented Mar 29, 2023 • edited by dontcallmedom

aboba left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aboba left a comment

Choose a reason for hiding this comment

jan-ivar left a comment

Choose a reason for hiding this comment

dontcallmedom-bot commented Apr 19, 2023

dontcallmedom-bot commented Jun 28, 2023

Orphis commented Mar 16, 2023 •

edited by pr-preview bot

henbos Mar 16, 2023 •

edited

fippo Mar 31, 2023 •

edited

dontcallmedom-bot commented Mar 29, 2023 •

edited by dontcallmedom

aboba left a comment •

edited