Section 3.2.1: Add N48 - bandwidth feedback speed

[xingri]

Partial fixes for #103

---

Preview | Diff

[aboba]

Issue #114 did not include text describing what aspect requires clarification. Since there is already an Issue open relating to Section 3.2 (#103), perhaps we can use that as the reference point.

The proposed Requirement N49 has been suggested as a problem for the Game Streaming use case previously, because players will be surprised and concerned when asked for camera and microphone permission in a game that does not make obvious use of those devices. So perhaps we should break N49 into its own PR.

The problem with support for rendering of "partial images" (e.g. slices or tiles) is that modern RTP payload specifications such as AV1 and VVC do not support SLI (Slice Loss Indicator), nor is SLI supported in any WebRTC browser. So there is no way to request recovery of a slice/tile, only the ability to negotiate support for NACK and RTX. This negotiation does not cover specific strategies like differential reliabilty (e.g. NACK only base layer), so there is no SDP semantics to work with. Is there specific functionality you have in mind?

While support for higher resolutions (e.g. 4K) and framerates is critical to game streaming, #103 points out that requirement N37 is not specific about what API changes are required. Also, N37 mentions copy removal which is more in scope for other WGs such as the MEDIA or WebGPU WGs than the WEBRTC WG. So currently PR #117 is reformulating N37 to remove mention of copy removal and to focus specifically on concerns raised by developers about lack of control over hardware acceleration in WebRTC compared with lower level APIs like WebCodecs.

For example, a 4K high framerate game would be likely to require hardware decode to function well. Since WebRTC is a high level API, it does not provide applications with information relating to hardware acceleration issues (e.g. inability to allocate hw resources or an error that causes failover to software). So a 4K high framerate game could failover to software, making players very unhappy, and there is no event or error that would provide this information immediately. Does PR #117 address your performance concerns?

[xingri]

Issue #114 did not include text describing what aspect requires clarification. Since there is already an Issue open relating to Section 3.2 (#103), perhaps we can use that as the reference point.

The proposed Requirement N49 has been suggested as a problem for the Game Streaming use case previously, because players will be surprised and concerned when asked for camera and microphone permission in a game that does not make obvious use of those devices. So perhaps we should break N49 into its own PR.

@aboba Thank you for the prompt review of the PR. I agree with you merging with #103.
And will split the PR for N49.

The problem with support for rendering of "partial images" (e.g. slices or tiles) is that modern RTP payload specifications such as AV1 and VVC do not support SLI (Slice Loss Indicator), nor is SLI supported in any WebRTC browser. So there is no way to request recovery of a slice/tile, only the ability to negotiate support for NACK and RTX. This negotiation does not cover specific strategies like differential reliabilty (e.g. NACK only base layer), so there is no SDP semantics to work with. Is there specific functionality you have in mind?

I believe you are referring this to N48. I will prepare more clarifications after internal discussion.

While support for higher resolutions (e.g. 4K) and framerates is critical to game streaming, #103 points out that requirement N37 is not specific about what API changes are required. Also, N37 mentions copy removal which is more in scope for other WGs such as the MEDIA or WebGPU WGs than the WEBRTC WG. So currently PR #117 is reformulating N37 to remove mention of copy removal and to focus specifically on concerns raised by developers about lack of control over hardware acceleration in WebRTC compared with lower level APIs like WebCodecs.

For example, a 4K high framerate game would be likely to require hardware decode to function well. Since WebRTC is a high level API, it does not provide applications with information relating to hardware acceleration issues (e.g. inability to allocate hw resources or an error that causes failover to software). So a 4K high framerate game could failover to software, making players very unhappy, and there is no event or error that would provide this information immediately. Does PR #117 address your performance concerns?

Thank you again for the feedback and agree with overall. But I also need to discuss this item internally.

[henbos]

I don't think supporting a certain frame rate or latency can be a requirement, because that depends on system, network and implementation.

Is it fair to say that the problem here isn't that "the implementation needs to be better!", rather that there is a tradeoff between more or less real-time use cases?

I'm wondering if the requirement should be phrased more around the app being able to influence the implementation to prioritize what is more important to the application?

[henbos]

What kind of API proposals would be addressing the cloud gaming use case? Are we talking about control knobs for the jitter buffer or forcing HW encoding?

[xingri]

@henbos thanks for the feedback. Will update the PR with more detailed clarifications for the cloud gaming.

[steely-glint]

I'm in favour of this, although it describes the mechanism rather than the outcome.
One extra possible knob on the jitterbuffer issue. There are situations where it is undesirable for the play out to speed up to catchup - (example is remote control of a vehicle where the perceived acceleration after a video freeze is very disconcerting.)
It would be useful to be able to set a flag that minimised this effect whilst still allowing the display of the most up to date frame.

[dontcallmedom-bot]

This issue was discussed in WebRTC July 2023 meeting – (PR #118: Clarify Game Streaming requirements (Section 3.2))

[dontcallmedom-bot]

This issue was mentioned in WebRTC TPAC 2023 meeting – 12 September 2023 (Low Latency Streaming: Game Streaming use case)

[xingri]

@aboba. I think the current issue has too many items for reviewing. So I am thinking about splitting this issue to 2 or 3 separate issues. How do you think about this idea?

[fippo]

Some of these are not W3C requirements but IETF ones.

RPSI is already covered by https://datatracker.ietf.org/doc/html/rfc8834#section-5.1.4

Controlling rtcp feedback frequency is similar to trr-int from RFC 4585. Specifically for TWCC you would need to extend the syntax similar to how RFC 8888 does it. I do think that sprang's suggestion to just use the newer (even less standard) v2 format which lets the other side control the interval through the header extension is better.

[aboba]

There are some existing discussions relating to LTR/RPSI:

IETF discussion of RPSI: aboba/hevc-webrtc#17
RPSI CL review: https://webrtc-review.googlesource.com/c/src/+/104880
(This review describes some of the problems in implementing RFC 7798-style RPSI. Not clear how SFU can determine whether
a P-frame based on the LTR would be decodable by participants. Issue is not just whether the LTR was sent, but whether it
was received, decoded and is currently stored in the buffer).

WebCodecs issues:
w3c/webcodecs#743
w3c/webcodecs#285

[xingri]

N51 is similar to trr-int from RFC 4585. Specifically for TWCC you would need to extend the syntax similar to how RFC 8888 does it. I do think that sprang's suggestion to just use the newer (even less standard) v2 format which lets the other side control the interval through the header extension is better.

@fippo Thank you for the feedback. Assuming you are referring to N50 instead of N51.

We understand that the Transport-wide congestion control 02 provides way to configure Transport wide CC feedback. However we want to have a API to set interval for Transport wide CC feedback to resolve the following concerns from the game streaming.

• This is not reliable as sender request might get lost in network resulting in no feedback.
• Sending extra headers results into reduced payload size.
• Sender request jitter shows up in feedback receive jitter.

[xingri]

N49 is already covered by https://datatracker.ietf.org/doc/html/rfc8834#section-5.1.4

There are some existing discussions relating to LTR/RPSI:

IETF discussion of RPSI: aboba/hevc-webrtc#17 RPSI CL review: https://webrtc-review.googlesource.com/c/src/+/104880 (This review describes some of the problems in implementing RFC 7798-style RPSI. Not clear how SFU can determine whether a P-frame based on the LTR would be decodable by participants. Issue is not just whether the LTR was sent, but whether it was received, decoded and is currently stored in the buffer).

WebCodecs issues: w3c/webcodecs#743 w3c/webcodecs#285

@aboba, @fippo Appreciate both feedbacks. We understand the current discussion.
So we want to pursue LNTF RTCP message as a short-term solution.
And we will continue to pursue on the RPSI approach and find a way to meet the codec agnostic concern raised by RPSI RTCP feedback support · Issue #13.

However, not only for the transport protocol topic, we want RTP de-packetization and framing would need to be updated to recover using non-key frame by the following reasons:

• Currently RTP receiver stops providing frames to decoder on packet loss.
• Need a way to start providing subsequent completely received non-key frames to decoder.
• Requires decoder API support (only encoder API discussed at TPAC)

[alvestrand]

Thanks for the split - I changed the title of both PRs so that it's possible to see which is which without memorizing the numbers.

[xingri]

Thanks for the split - I changed the title of both PRs so that it's possible to see which is which without memorizing the numbers.

Likewise, thank you for the title changes and feedback. Will prepare updated sentences for this after internal discussion.

[xingri]

actually not, because that's not how it works in most CC algorithms (REMB is an exception) - a media receiving user agent will tell the media sender user agent about arrival times of packets (or, when delay-based mechanisms are not used, just which packets have arrived), and the sending agent computes its estimate of throughput based on that.

State your requirements, don't constrain the mechanisms.

(Note that as stated, the requirement would be fulfilled by having a control knob that only delayed notification to the application about bandwidth changes - you haven't stated either the direction you want to push it in or the amount of pushing you want to be able to apply.)

@alvestrand, @stefhak I have updated the requirement, please review the updated sentences.

[stefhak]

@xingri the current version looks good to me (with a comment similar to the one provided for #129: the part after the comma is more of an explanation and should perhaps be removed, or moved to the use case description, or made into a note).

[xingri]

@xingri the current version looks good to me (with a comment similar to the one provided for #129: the part after the comma is more of an explanation and should perhaps be removed, or moved to the use case description, or made into a note).

@stefhak Thank you again for the feedback. Remove the sentence after comma.

[xingri]

@alvestrand & @aboba Could we think we reached the consensus for this requirement? If so, could we merge this requirement? Otherwise, we would like to discuss this on upcoming working group call on Jan-16.

[xingri]

@jan-ivar Thanks for sharing the feedback during today's call. I am more than happy to update this PR by your feedback during the call but would you mind sharing your feedback once again on this PR?
Then it will be more helpful for me to get the same consensus as you shared today.

[dontcallmedom-bot]

This issue was discussed in WebRTC March 26 2024 meeting – 26 March 2024 (PR #118 Bandwidth feedback Speed(Configurable RTCP transmission interval))

[xingri]

@jan-ivar could you please review the updated message?

[aboba]

Rather than requiring the application to directly set timing parameters (which could enable exploits or conflict with UA CC algorithms), the requirement could be formulated as "allowing the application to provide its timing parameter preferences to the UA". Instead of trying to influence sets of timing parameters, is there a way to boil the feedback down, such as a via "Feedback Timing Hint" with values of "default", "rapid" or "delayed"?

With "Content Hints" we ended up using a DOMString instead of an enum to provides the UA with additional information on the content of a MediaStreamTrack, but that is just a detail.

[xingri]

Rather than requiring the application to directly set timing parameters (which could enable exploits or conflict with UA CC algorithms), the requirement could be formulated as "allowing the application to provide its timing parameter preferences to the UA". Instead of trying to influence sets of timing parameters, is there a way to boil the feedback down, such as a via "Feedback Timing Hint" with values of "default", "rapid" or "delayed"?

With "Content Hints" we ended up using a DOMString instead of an enum to provides the UA with additional information on the content of a MediaStreamTrack, but that is just a detail.

Thanks @aboba for the feedback. I have updated sentences. Could you please review it?

[xingri]

@aboba, @alvestrand and @jan-ivar, as we discussed at the last working group call, could you please review this PR and share feedback? If there are no any further changes required, could we submit this as a game streaming requirement?

[stefhak]

I like the idea of allowing the application to hint to the user-agent its preferences regarding feedback speed. The current text IMHO looks more like an explanation than an actual requirement though. Could we have an requirement phrased more like "The application must be able to indicate its preference regarding feedback speed to the user-agent." instead?

The actual phrasing aside, going in this direction looks quite similar to the degradationPreference we have on the RTP-sender. It is basically a hint to the user-agent on what to do if the network capacity is limited. I think a similar thing could be done to the RTP-receiver, with a feedbackSpeedPreference. As the feedback would compete with audio/video (for bidirectional services - for gaming this is not the case) in the reverse direction it is actually quite similar.

Anyway, where and how to API this is a later question, it would be good to get agreement on requirement phrasing now.

[alvestrand]

I'm afraid we're still not writing requirements. And when we're just modifying the requirements, not the text of the scenario, it's a sign that we're not writing the right thing.

In this case, the problem we have is that stalls on a game stream (in-game, not just twitch-like) is much more problematic than stalls on a media-watching or even videoconferencing stream - because it doesn't just irritate the players, it makes players lose. It's not possible to read out of the current text that this is the problem we're trying to solve.

The first part of the requirement is actually fine. The second is an implementation suggestion.

I suggest adding to the scenario description, after "game players not reveal their location to each other", a sentence:

"Stalls and variations in latency in game data streaming are highly disruptive to gameplay. Preventing those things therefore has a higher priority than in conferencing or media-watching use cases."

I'd also suggest dropping the text from "A Feedback Timing Hint" on out, because that describes a mechanism, not a requirement.

[stefhak]

I agree to what @alvestrand says in #118 (comment).

[alvestrand]

(note: for preventing stalls and variations in latency, a possible remedy is support for L4S. I don't know if we need API for this, since at the moment I can't see a reason to turn L4S off, once it's properly implemented. So this may be an IETF-only concern.)

[stefhak]

(note: for preventing stalls and variations in latency, a possible remedy is support for L4S. I don't know if we need API for this, since at the moment I can't see a reason to turn L4S off, once it's properly implemented. So this may be an IETF-only concern.)

I also cannot see a reason for turning it off. I wonder if there would be any value for the app to know if it's being used or not. But for L4S we're in the mercy of network and connectivity providers, so the requirement we're discussing here is definitely needed.

[xingri]

@alvestrand Just updated the sentences to have the first part. Please review it and let me know your feedback.
FYI: @aboba, @stefhak Please

[stefhak]

I think the requirement looks ok now. I would slightly rephrase (rather than saying the "application should be designed" I'd say something like "application must be able to"). But let's see what others think before changing.

I do think you should add the text "Stalls and variations in latency in game data streaming are highly disruptive to gameplay. Preventing those things therefore has a higher priority than in conferencing or media-watching use cases." in the scenario description after "game players not reveal their location to each other" as @alvestrand proposed in #118 (comment).

Finally, it is easier to review changes if you keep all commits :) they can be squashed when merging to main (which I hope we will get to soon...)

[aboba]

@alvestrand L4S operation requires ECN feedback, which may imply support for additional SDP parameters. For example, RFC 8888 Section 7 says:

"When congestion control feedback is to be used with RTP and ECN, the SDP offer generated include an
"a=ecn-capable-rtp:" attribute to negotiate ECN support, along with an "a=rtcp-fb:" attribute with
the "ack" parameter "ccfb" to indicate that the RTP Congestion Control Feedback Packet can be
used."

Requiring SDP munging to implicitly select congestion control algorithms seems undesirable. It's also not clear to me that we need much in the way of API surface, other than perhaps a configuration parameter that will generate the required SDP parameters by default. But we should be cautious about API changes because once L4S has been deployed and has been sufficiently tested it could become the default, so that the configuration parameters might be less needed.

[aboba]

The packet loss issue is somewhat distinct so it could have its own requirement. One of the issues encountered in mobile gaming is that loss may not be congestive, but could be due to wireless issues such as fading or poor rate adaption. There have been some papers on distinguishing congestive from non-congestive loss. In a gaming scenario, the optimizations would typically be handled on the server, so it's not clear to me what changes would be required in the WebRTC-API.

On the congestion control/timing side, part of the reason for major variations in the bandwidth estimate is queue building that both increases the latency of the game and causes variability in bandwidth estimates when the queues overflow. Increasing the frequency of feedback can help in riding these waves, but in the long term we can smooth out the waves, achieving both lower packet loss and smoother variation in bandwidth estimate via congestion control algorithms such as L4S.

[stefhak]

@aboba said:
"In a gaming scenario, the optimizations would typically be handled on the server",
Exactly, and for the server to be able to react/act/optimize as quickly as possible there is a need for the UA to send feedback ASAP when condition changes are observed. This was discussed on the call in March (https://www.w3.org/2024/03/26-webrtc-minutes.html#t08), and my understanding is that many see the need for being able to ask the UA to send feedback quickly (even if that would mean overshooting the 5% RTCP limit (which would not matter much in the gaming scenario as there would be no video in the reverse link)).
"so it's not clear to me what changes would be required in the WebRTC-API." In my view it would be a hint or something similar, 'please optimize for fast congestion/channel changes feedback'. But the actual API can be discussed later, I do think the requirement for some kind of API should be landed now.

Regarding the discussion on wireless channels, I think modern radio links have FEC and radio link retransmissions. So even if the problem is coverage, there would be a gain in reducing the send rate as more FEC could be applied and fewer retransmissions (introducing congestion in a sense for the radio hop as it is a shared medium) would be needed.

I agree, in a longer term L4S will help a lot.