Video h264 decoder supports both Annex B and AVCC format

[daijh]

Both are very popular and used in different scenarios. Let application to choose the format during decoder configuration.

[sandersdan]

AVC format:

• Used in media containers
• Configuration up front (avcC), parsing can be generic
• Packet headers have lengths, efficient to parse and validate

Annex B format:

• Used in streaming protocols
• Configuration in-band, parsing must be codec-aware
• Packets have end markers, less efficient to parse and validate

For WebCodecs, the configuration difference is probably the most important. The current design expects all configuration changes to be explicit; if we require apps to detect and extract Annex B configuration changes, then we may as well require them to convert to AVC format at the same time.

Other options include supporting Annex B but rejecting streams with in-band configuration changes, or supporting Annex B with a new configuration model.

[daijh]

Most typical Annex B in-band configuration change is video resolution, for adaptive bitrate streaming. Usually video decoders are able to handle it silently.

If unlucky the in-band configuration change was not supported by decoder (perhaps profile change), throw an exception for re-configure decoder and continually decoding with new configuration. This case should be very seldom.

[chcunningham]

Latest thinking is we should add some field to the config to describe which format is desired.

[OllieJones]

For what it's worth, the MediaRecorder in Google Chrome generates Annex B style data streams.

When asked to generate MIME types like video/mp4; codecs="avc1.42E01E", it generates webm data streams. The h.264 data is in a series of Cluster / Simple block frames in the Matroska container.

Each of those simple blocks contains a sequence of complete AVC NALUs (sometimes just one, sometimes several). They're separated by Annex B style 0000 0000 0000 0001 start codes rather than avcC style lengths. (The first frame in each webm Cluster starts with SPS and PPS NALUs, then an intraframe instantaneous decoder refresh picture NALU.) They are complete NALUs, so there's no need to piece together consecutive frames.

It's not hard to convert the start-code format to the byte-length format. But it would be nice if the decoder ate both.

[chcunningham]

For those following along, the linked PR #117 solves format selection for encoding, but not decoding.

For decoding, my thinking is to simply signal your preferred format by including/excluding the avcc via the VideoDecoderConfig.description. No avcc implies annex-b format.

[daijh]

Agreed it is not hard to implement Annex B to avcC conversion in js. I have already implemented one and works well.

As @sandersdan suggested previously, how does the decoder handle adaptive video streaming (dynamic resolution) properly with these two formats? Does WebCodecs allow silent configure change?

• If it is Annex B, the decoder should detect the configure change and notify user?
• If it is avcC, every time the resolution changes, user need flush decoder, reconfigure decoder, and start decoding with new configuration. Not sure is that good for latency sensitive usage like cloud gaming?

[sandersdan]

My expectations are:

• For Annex B, the WebCodecs implementation will detect and handle configuration changes silently. (I recommend making sure all PPS units are available before the preceding keyframe, not all of Chrome's decoders can change to a previously unknown PPS mid-GOP.)
• For avcC, apps must reconfigure, but they do not need to flush or wait. Configuration changes are simply queued along with chunks to be decoded.

[daijh]

For avcC, apps must reconfigure, but they do not need to flush or wait. Configuration changes are simply queued along with chunks to be decoded.

The VideoEncoder and VideoDecoder reconfiguration is expected to be the same procedure. It will be good if VideoEncoder also supports reconfiguration in this way.

Options are metadata required to construct a compliant bitstream (for example, the codec name and profile). These are required when configuring the encoder/decoder and cannot be changed without again calling configure(). Reconfiguration will internally reset the codec, aborting any pending outputs. Callers can avoid having outputs aborted by first flushing (flush()) the codec.

https://github.com/WICG/web-codecs/blob/master/explainer.md#codec-configuration
The description in explainer is also needed to revised.

[sandersdan]

It will be good if VideoEncoder also supports reconfiguration in this way.

It does.

The description in explainer is also needed to revised.

@chcunningham

That's fair, the explainer has not been kept up-to-date with changes in the draft spec.

We should probably just cut a lot of detail out of it and refer to the spec instead.