Decoding "out of order" video frames (B-frames)

[steveanton]

Many video codecs support B-frames, frames which have dependencies on previous and future (chronological) frames. Some decoders want to be fed packet in the decoder order (e.g., the future I-frame is fed before the B-frame that depends on it) while others will take care of buffering and reordering internally.

This is a tracking issue for sketching how the WebCodecs API supports B-frames.

[mfoltzgoogle]

Why does WebCodecs need to do anything? In the same implementation, is it likely that the encoder for a B-frames codec (say h264) would generate a bitstream that the decoder wouldn't be able to handle?

I suppose there are some implementations with software encoders and hardware decoders. Is that what you were thinking?

[pthatcherg]

It impacts two things:

1. The API of the decoder. If B-frames are handled internally, I believe you need to pass a timestamp into the decoder letting it know the order things should come out in. I don't believe that info is in the bitstream, but I could be wrong. Also, you need to know how many frames will be buffered to know how much internal delay there is (how many frames you need to give it before frames come out the decoded side).

2. The API of the encoder. You need to control if you want B-frames or if you want a low-latency encoder, and perhaps how far the the B-frames can reach (impacting the amount of buffering required by the decoder).

3. Capabilities for the encoder and decoder related to 1 and 2 above.

[ytakio]

I used to implement and develop H.264/AVC software encoder at my former job.
So please let me comment from its perspective.
(I'm sorry that my comment doesn't take care of other codec at all)

1. The API of the decoder. If B-frames are handled internally, I believe you need to pass a timestamp into the decoder letting it know the order things should come out in. I don't believe that info is in the bitstream, but I could be wrong.

A decoder doesn't need a timestamp for reordering frames which includes B-frames. Because each coded frame has Picture Order Count which represents output order in each slice header. So it is ok for decoder just to be pushed the Access Unit in decoding order, then you can get each decoded frame in output order.
Additionally, encoder and decoder basically don't treat a duration of each frame. (You can describe frame rate in VUI, though.) While to play correctly needs timestamp and duration. I think a container should take care of it. So decoder/encoder API should be able to pass through this information with target frame.

Also, you need to know how many frames will be buffered to know how much internal delay there is (how many frames you need to give it before frames come out the decoded side).

I'm not sure about decoder implementation in detail. But I think decoder could output immediately a frame if it is output order. Anyway, container should take care of timing for decoding and presenting.

2. The API of the encoder. You need to control if you want B-frames or if you want a low-latency encoder, and perhaps how far the the B-frames can reach (impacting the amount of buffering required by the decoder).

I think it depends on the encoder capability. Encoder will decide which frame type should be used in the end. (e.g. IDR:IDR, I:I, P:I/P, B:I/P/B)
So if user want a low-latency stream, then they set just IPPP sequence, I think.
And picture decoded buffer size in decoder is specified by Profile and Level.

[sandersdan]

The main limitation we have here is that not all platform decoders have the option to output frames in decode order. The common denominator then is that WebCodecs video decoders should output frames in presentation order. Some implementations would need to buffer out-of-order frames to make that happen.

[murillo128]

I think that in case we support b-frames in both encoder/decoders, the frames should be output in encoding/decoding order and have a TransformStream that reorders them

12345->Encoder-> I(1)P(5)B(2)B(3)B(4)->Decoder->15234->transformstream->12345->mediastream

We should add pictId to the VideoFrame so the reordering is possible.

[ytakio]

I think that it is difficult to take a presentation number from decoding order frames. It will not be able to find a number of frames between reference frames...
So a decoder WebCodecs uses should take care of it by using information in raw stream.
Or if a decoder can label a pictId to the VideoFrame, it'll be good, too.

A limitation size of buffering out-of-order frames must be specified by each codec specification.

[chcunningham]

As others have pointed out, #55 is basically a duplicate of this, but with more current insights and more recent feedback. Closing this one in favor of that.

The API of the encoder. You need to control if you want B-frames or if you want a low-latency encoder, and perhaps how far the the B-frames can reach (impacting the amount of buffering required by the decoder).

This is tracked in greater detail by #57