VideoDecoder H.264 decode is sometimes one frame short

[jamespearce2006]

I'm decoding individual H.264 GOPs and I'm finding that sometimes the decoder will not emit the last frame, and I'm not sure what I'm doing wrong. It's not a problem with the source data as the same GOP will work and then fail. I'm assuming it's something wrong with the way I'm using flush.

The GOPs are 25 frames long, and are closed.

I'm calling decode() for each sample in the GOP (so 25 times), then calling flush(). On each callback, I generate an imagebitmap from the videoframe, then call destroy on the videoframe.

Most of the time it all works OK, and my output callback is called 25 times. Occasionally, My callback is only called 24 times and the final frame is not emitted.

Can anyone think what I might be doing wrong? Or could this be a genuine issue?

I have found that I can apparently fix the issue by calling flush() every time my frame callback is called, which seems to add weight to my thought that this is a flush problem. The solution seems a bit expensive though.

[tguilbert-google]

The output callback can arrive after the flush promise resolves. Calling flush between each decode call is not recommended. Does creating a promise that resolves when #outputs matches #decodes and awaiting both decoder.flush() and that promise solve the issue?

[jamespearce2006]

It doesn't seem to make any difference having a promise that resolves when the flush completes AND the expected number of frames is emitted.
There seems to be a genuine issue that flush doesn't always push out all of the decoded frames. That frame does arrive with the next GOP, so it's not that the frame is not being decoded, it's just that flush isn't always working reliably. Unfortunately, I'm decoding GOPs individually, so I need all of the frames.
The only way I can get all of the frames reliably is to repeatedly call flush on receipt of every decoded frame.

I've also noticed that flush doesn't work properly with AudioDecoder. In fact, calling flush seems to stop the decoder from outputting samples altogether. It works fine without the flush though.

[chcunningham]

Hey @jamespearce2006 - agree this sounds like a bug. Are you able to provide a script to help us reproduce the issue?

[jamespearce2006]

I've put some sample code here:
sample.zip

This sample repeatedly attempts to decode the same GOP. I'm using an RxJs Observable that will emit on receipt of a frame, and complete once the decoder flushes AND all 25 frames have been emitted.

When I run this, it usually fails (the observable times-out) after just a handful of GOPs. It's usually stuck at either 17 frames or 24 frames.

[sandersdan]

I have not attempted to reproduce yet but I did want to add two points:

• flush() should only be called at the end of a GOP. We don't currently guarantee that we can decode a non-keyframe after a flush().
• It would be an implementation bug for WebCodecs to complete a flush promise before outputting all frames. This is one of the reasons that output is a callback--it's less likely that apps will inadvertently reorder them by chaining promises.

This should probably be filed as a Chromium bug rather than a spec issue where it will be more appropriate to start digging into the details.

[jamespearce2006]

When you say "at the end of a GOP", do you mean once decode has been called on all frames from the GOP? Or when decoded frames have been emitted?
What I'm attempting to do is call decode() for each frame in the GOP, followed immediately by a flush(). This doesn't seem to work reliably.
The only thing I've found to work reliably is to actually call flush() in my callback every time it is invoked, which seems like a monstrous hack.
I will open a Chromium bug if it doesn't seem like I'm abusing the API.

[sandersdan]

Your description sounds correct, flush() should be called at most once per GOP, after calling decode() for each frame. In Chromium's implementation, flush() is only a synchronization primitive, our decoders are not supposed to hold on to frames indefinitely anyway, so flush() is a sort of end-of-stream marker. A conformant WebCodecs implementation could require flush() to emit the last few frames, though.

I would not expect applications to call flush() after every GOP unless they are very concerned about the latency before they start decoding frames from the next GOP.

[jamespearce2006]

@sandersdan In my case, I need to call flush() on each GOP because I'm not decoding GOPs for linear playout. I'm typically scrubbing or doing odd play speeds, so I need to be able to acquire the frames for a single GOP before moving on to a different GOP.

BTW I finally got round to opening a Chromium bug:
https://bugs.chromium.org/p/chromium/issues/detail?id=1168572

[sandersdan]

Thanks! I'll close this and we can continue there.