presence of composition time offset in PacketTypeCodedFrames is codec-specific

[zenomt]

in the VideoTagBody section of version 2023-03-v1.0.0-B.9, there are two PacketTypes defined for coded frames: PacketTypeCodedFrames = 1 and PacketTypeCodedFramesX = 3, where type 3 has an implicit composition time offset of 0 (meaning the presentation time and decode time are the same).

the pseudocode for If FourCC == HEVC explicitly shows the composition time offset SI24 for PacketTypeCodedFrames and defines it to 0 for PacketTypeCodedFramesX. the other defined codecs (VP9 and AV1) use PacketTypeCodedFrames but don't show a composition time offset (presumably because those codecs don't code independent frames out of presentation order so such a field isn't needed).

since the notion of a presentation time different from decode time occurs in at least two codecs today (AVC and HEVC), and in the interest of consistency, separation of layers and concerns (notional header vs payload), and code reuse, i think PacketTypeCodedFrames should always have an SI24 Composition Time Offset field, and for codecs where that's always 0, they MAY use PacketTypeCodedFramesX to save those 3 bytes. any codec should be able to use either packet type 1 or 3 with a consistent parsing and interpretation.

ideally the Enhanced RTMP spec wouldn't itself define any new codec mappings, and instead would define the generic syntax. an addendum, appendix, or separate registry would define mappings for AV1, VP9, and HEVC to start.

[veovera]

Makes sense. I should be able to reword the spec for this item without breaking 2023-03-v1.0.0-B.9 behaviour.

[zenomt]

without breaking 2023-03-v1.0.0-B.9 behaviour

i think this would be breaking for VP9 and AV1, since PacketTypeCodedFrames for those codecs doesn't have the three bytes of composition time offset currently. certainly adding support for PacketTypeCodedFramesX for those codecs would be forward compatible.

[veovera]

Really good feedback. For now we will leave as is to not break currently defined behavior. When we add new codecs we may adopt the generic syntax for newly adopted codecs. Closing the issue