Receiver transform depacketizer requirements

[youennf]

The depacketizer is receiving packets and is concatenating packets into frames.
Frames may be out of order if some packets are reordered.

The decoder is processing frames and they must be ordered appropriately.

The receiver transform sits in the middle between depacketizer and decoder.
It is unclear whether the reordering happens before transform or after transform.

Given this is visible to the transform, the spec should be clear about this.
From the current wording, the frame timestamp should be monotonously increasing so it seems to indicate reordering should be done before frames get exposed to the transform.

On encoder side, it could also be possible for encoder to reorder video frames but most encoders probably do not do that.
It would still be good to clarify that encoder order is sender transform order and depacketizer should probably make sure this order is respected for receiver transform (and thus decoder).

[alvestrand]

The bogeyman here is (of course) H.264 B-frames, which will cause frames with earlier timestamps to be sent (and received, if in order) after the (future) frames they refer to.

The RTP sequence number is monotonically increasing in depacketization order; this argues for exposing the RTP sequence number and requiring that the frames are presented to the transform in RTP sequence number order.

@guidou might have the reference.

[youennf]

Editor meeting: consensus to specify that sender side should be enforced at sender and receiver side.

[alvestrand]

This turns out to be unexpectedly problematic for the receiver side.

The internal logic of frame processing leaves the reordering as late as possible, just before the decoding step, which occurs on a relatively high priority thread so that lost and retransmitted packets have the maximum time available for catching up and filling gaps.
But if we insert JS processing at that point, it means that:

a) it tightens the delay budget for the transform considerably; any jitter in processing time will be directly reflected in jitter in display
b) if one desired to write one's own jitter buffer in WASM, waiting for out-of-order frames before the WASM processing would defeat the whole purpose of that function.

So I'm reopening the issue; let's discuss more.