Synchronize audio/video with data in WebRTC

[tidoust]

Some potential use cases for WebRTC presented during last WebRTC WG F2F would require a mechanism to synchronize audio/video with data.

One possible approach would be to use Real-Time Text to timestamp data with the audio/video, but the question of the synchronization of that stream with the audio/video streams remains to some extent (depending on the use case and precision required).

Some questions to handle:

1. What is the browser's role in the synchronization? Should it sync things up on its own? Is it enough to expose some info and knobs such as the relationship to the Performance.now clock and the average latency of the processing pipeline, as done in the Web Audio API?
2. If processing of the data needs to be done by the app, what does it mean to synchronize the streams? I.e. would triggering an event at the right time be enough when high-precision is required? Or should the app rather have a way to monitor the information one way or the other through a worklet, or a processing mechanism close to the rendering pipeline such as requestAnimationFrame?

It is interesting to see the relationship between this need and similar needs expressed over the years by media companies. For instance, the Timing on the Web effort (#46) would help solve the first point, and was not triggered by WebRTC use cases. Also the second point was recently discussed at length in the Media & Entertainment IG (see w3c/media-and-entertainment#4).

[dontcallmedom]

Early discussions on related topics at the June 2018 WebRTC F2F meeting

[dontcallmedom]

The topic was mentioned during the recent media production workshop.

The overall direction is tracked as a WebRTC NV use case, but could use more direct involvement from potential users to get more momentum.

[darkvertex]

The overall direction is tracked as a WebRTC NV use case, but could use more direct involvement from potential users to get more momentum.

Hi! 👋 I've been lurking on here. Thought I'd share you a first hand VR-related use case where it could have been very useful:

We needed to deliver N concurrent synced video feeds from a multi-lens VR camera rig from a location with poor computational capacity (too low for live-stitching panoramas onsite.) For reasons I cannot disclose we needed to livestream 360 video with a VR camera that wasn't able to livestream a stitched 360 video natively. The workaround we decided was to receive the individual feeds elsewhere with a more powerful computer and produce the stitched 360 monoscopic panoramic video to stream to whatever. (You can conceptualize each lens feed as an RTP/WebRTC video track.)

Our camera had 8 physical lenses horizontally but with just 4 we already had sufficient panoramic coverage so to save some bandwidth we only send 4.. but which 4? Depends what's visible and near what lens; maybe we want 4 even or 4 odd. We designed the sending software to let you pick a subset of the cameras. We can switch which are active on a whim, mid-stream.

One approach to dynamic feed switching in WebRTC could be to prenegotiate all the tracks possible that you could need and only send video on those that you consider active, but it's a little tricky to distinguish between a video feed being suddenly inactive because it was intentionally disabled due to a reconfiguration at the sender VS a video feed being suddenly inactive because there was some network congestion / data loss down the pipe. Renegotiating WebRTC video tracks between configuration switches is possible but we felt it interrupted the flow considerably as it added some overhead, so we didn't go with it.

We needed the camera configuration and identity metadata to be timestamped with the video frames so it could correlate in perfect sync during reconfigurations. (Feed 1 may be showing camera 0, but maybe five seconds later it's showing camera 1, for example.) Identity matters for a realtime panoramic stitch because they are different perspectives in space and the algorithm must be kept informed or it'll look wonky. Unsynced changes are not useful as it glitches the result of the 360 processing and a WebRTC data channel (to my knowledge) could not do this with today's WebRTC generation.

We absolutely needed the camera identity to be in sync with the video frames. Since WebRTC data channels fell short, we simplified further down and settled on a pure RTP approach. We opted to hijack the outgoing H264 bitstream and inject NAL units of type SEI (Session Enhancement Information) Subtype 5, aka "unregistered user data" SEI , in with the frame data in the data of the RTP video track. You can slip small amounts of userdata (text or json or whatever) in the video feed this way and not corrupt anything. All video players safely dismiss it.

On a custom software receiver (not a regular browser) you can recover the H264 packets from the RTP track, recover the original NAL units and read the metadata, and your video processing can react accordingly since the metadata changes in sync with the video frames.

---

If data channels could be in sync with video without codec gymnastics, or if another convenient mechanism existed for a generic timestamped metadata stream, I think we may have stuck with WebRTC for our use case. (I personally would have liked that as it could have made it easier to debug things from a web app in-browser instead of some custom standalone software.)

Ultimately data being in sync with video is important to any kind of realtime actor with a need for a status HUD:

• Imagine a FPV drone web app where you can control it and there's a HUD overlay showing the gyroscope data in sync with the video,
• or one of those creepy walking robot dogs and there's charts graphing the servo rotations and you can see exactly when one of them jams because it's in sync with the video showing you the same.

Sending device health and state information in perfect sync with the video feed is crucial for a trustworthy assessment of what's happening on screen of a remote entity. Being able to do this in an official and reliable capacity would be exciting!

---

By the way, I took a superficial look at the WebCodecs API but it seems it's very focused on decoding video images and not so much on letting me see individual NAL units of an H264 bitstream. (To those familiar with it: Is this a fair observation or did I miss something?) It sure would be cool if it had some callbacks to access non-video/audio metadata.

Sorry for the wall of text. Hope it helps shed some light on why synced data could open the door to some very handy in-browser use cases.

[dontcallmedom]

@darkvertex this is amazing input, thank you so much! Could I ask you to bring it to the WebRTC NV use case repository as a new issue (e.g. titled "detailed example of value of A/V/data sync in WebRTC")?

This would make sure others in the WebRTC WG see it, and hopefully respond with insights and questions.

[chrisn]

It sure would be cool if it had some callbacks to access non-video/audio metadata.

This relates to w3c/webcodecs#198, and a recent proposal to add API support for SEI events presented to M&E IG (minutes), although that focuses on the HTML video element. We're currently investigating whether this aligns with the DataCue proposal. This seems like another area where cross-group discussion could be helpful.

[darkvertex]

@darkvertex this is amazing input, thank you so much! Could I ask you to bring it to the WebRTC NV use case repository as a new issue (e.g. titled "detailed example of value of A/V/data sync in WebRTC")?

Thank you! I didn't know that other repo existed; thanks for directing me. I made the issue:
w3c/webrtc-nv-use-cases#74