Handling of encoding/decoding errors

[youennf]

In some platforms, the encoding/decoding capacities are limited.
It may happen that encoding/decoding frames will fail due to encoder/decoder being not available.

This error might be raised at various time, including during an established session, after a call to applyConstraints for instance which will change the size of the frames.

There does not seem to be a way to convey that information to web applications right now.
Would it make sense to add some error mechanisms there, for instance at sender/receiver level?

[aboba]

With respect to the resource model Section 4.3.2 says:

"If a system has limited resources (e.g. a finite number of decoders), createOffer needs to return an offer that reflects the current state of the system, so that setLocalDescription will succeed when it attempts to acquire those resources. The session descriptions must remain usable by setLocalDescription without causing an error until at least the end of the fulfillment callback of the returned promise. "

I think your question relates to what happens once media is flowing and the situation changes (e.g. CPU or encoder/decoder resources become scarce). The model is "best efforts" where the encoder/decoder will do the best it can based on potential guidance (e.g. for the encoder, degradationPreference.). Rather than resulting in an error, performance (which can be monitored via WebRTC-stats) will degrade. The application can monitor this and adjust parameters to try to improve the situation.

[youennf]

I am not exactly sure what 'performance degradation' means.
In some cases, encoding/decoding may totally stop.

In that case, getStats could be used to detect when encoding/decoding is failing.
But it seems weird to do continuous calls to getStats just for that reason.
It also does not give any hint to the web application of the actual error and might not be very responsive.

If an error event was sent (or something equivalent like an error callback) with some reason of the error, it would be much simpler/quicker/easier for the web application to react upon it.

[aboba]

@pthatcherg @fluffy Any thoughts?

[aboba]

This issue was discussed at the virtual interim on June 28. Consensus was that the specification utilizes a "best efforts" model, so while performance can degrade as reflected in stats, an onerror EventHandler is not needed in the RtpReceiver/Sender.

[youennf]

I won't object since getStats is a kind of workaround here.

That said, I don't think we should rely too much on the web engine "best effort" model here.
Web apps can usually take actions that have bigger impacts on performance and usability than what a web engine can do. Changing the resolution of a video track at the source is better than dropping the resolution or dropping frames later in the chain. If no encoder slot is available, the web application may decide to drop a low priority track.

Also, there are on-going discussions to expose things like memory pressure or cpu overuse to web pages. WebRTC applications could benefit from that, especially if WebRTC engines could provide some more feedback. getStats is the current answer but there might be room for further improvement.