More WebRTC related fields.

[ilyanikolaevskiy]

One still missing field in the VideoFrameMetadata dictionary is remote capture time (e.g. "int64 remoteCaptureTime".

The rationale is as follows: in WebRTC scenario the video originates at a separate machine, which has a separate clock. There might be clock offset or even drift.

WebRTC estimates capture time in both local and remote time-bases. It's logical, that capture time in local clock is already intended to be in captureTime field. This can be used to measure E2E delay.

But the same capture time in remote clock is very useful for synchronization purposes. Suppose the application wants to e.g. render emojis on top of the video or show chat messages, synchronized with the source video. The send side can easily obtain current local clock and attach that as a label to the chat message. Then receiving side could use remoteCaptureTime to decide if this is the frame suitable for showing the message, because they would be in the same timebase.

As this remoteCaptureTime is not in local clock it doesn't even have to be a timstamp type, just some monotonically increasing number, presumably in milliseconds at the send machine should work.

This field will duplicate rtpTimestamp a little, but it's quite different: it's not present for all the frames, unlike rtpTimestamp; It doesn't wrap and can be simply compared with the labels associated with a extra synchonizable content.

[tguilbert-google]

Yes, this seems like a good addition, and you are not the only one interested in this field.

Adding the field to the video-raf spec and implementation is easy, but I'm less familiar with WebRTC implementations. Does WebRTC already have a spec'ed way of capturing the remote time and sending it over?

[ilyanikolaevskiy]

Yes, all the frames have RTP timestamps, and RTP standard dictates how to transform these to the NTP capture time (in a sender clock) via RTCP SR messages:
https://tools.ietf.org/html/rfc3550#section-6.4

RTCP SRs have corresponding NTP and RTP timestamps. Given at least 2 sender reports the receiver can estimate corresponding NTP time for any given RTP timestamp.

[tguilbert-google]

This has been added to the spec, and landed in Chrome.

Thanks!