Encode and Decode error event

[drkron]

I'm looking for a way to detect decoding errors (both in the sense of failure to decode a frame and also in the sense that the decoder is falling behind due to scarce resources.

My suggestion is to add error events for decoding and encoding errors.

This was also proposed two years ago in #1323, where the consensus was that getStats() would be good enough.

The primary target for me is to use this to detect when the decoder is falling behind.
A concern I have with using getStats() is that it would require the user to have some custom heuristic code to detect this. Since this would be based on a rather limited set of information it would be noisy and it also requires the user to constantly pull getStats() for information. I believe that a signaling event that is fired from within WebRTC should be possible to make more accurate and robust.

Another problem is that some HW encoder/decoder uses a pipeline so that there's always a one-frame delay. In this case a high encode/decode time does not mean that it's falling behind. In such a case a detection in WebRTC could look at how queues are building up.

[aboba]

I am wondering if this issue applies only to webrtc-pc or perhaps also to media capture. A device (such as a webcam) can have a built-in encoder. Sometimes the device will successfully initialize, but later on there will be transient encoding errors, or perhaps encoding will stop.

There is a DOMException with a name of EncodingError that we could use, but it seems like having some extended information (as would be possible with RTCError) might be more useful.

[aboba]

@drkron Would you be available to present this issue (and potential solutions) at the July 2 Virtual Interim?

[drkron]

@drkron Would you be available to present this issue (and potential solutions) at the July 2 Virtual Interim?

I'm on vacation but I will try to attend the meeting.
Is there an agenda for the meeting, that is, is it decided in which order all issues are discussed?

[henbos]

Is there an agenda for the meeting, that is, is it decided in which order all issues are discussed?

Yes, see the slides. I gave you write-access with your google account

[aboba]

Note that there is an RTCError defined for a hardware encoder error in Section 5.2 (used by setParameters):

"If an error occurred due to a hardware encoder not supporting parameters, reject p with a newly created RTCError whose errorDetail is set to "hardware-encoder-error"

[drkron]

I'm looking into using the existing stats, framesReceived and framesDecoded, to detect if the decoder is too slow. From my initial testing it seems as taking the difference gives a quite good indication of the current decoding status.
So for now I don't see the immediate need for this feature.

[aboba]

@drkron Can we close this issue?

[henbos]

Maybe this will be addressed by WebCodecs?

[drkron]

From my point of view, it can be closed.