[WebCodecs VideoFrame metadata registry] Introduce VideoFrame metadata

[youennf]

---

Preview | Diff

[youennf]

@sandersdan FYI.

[sandersdan]

We just discussed this among the Chrome media team, and it turns out we're okay leaving out the user metadata. This would allow us to sidestep most of the questions about serialization.

The reasoning here is that if we're not storing metadata also on chunks, and we're not copying metadata between chunks and frames, then users will have to map their own metadata to frames separately anyway (presumably by timestamp). Therefore the user metadata is just a convenience rather than a critical feature in the short term.

[youennf]

it turns out we're okay leaving out the user metadata.

FWIW, one use case I can see is the AR/VR use case where metadata is retrieved by a worker through WebRTC encoded transform, then metadata is passing through decoder to MediaStreamTrack VideoFrames. In that case there might be two contexts so it seems nice to attach application metadata to VideoFrames directly.

That said, I am more than happy to leave this to another day/another PR.

[youennf]

I removed user metadata from this PR.
I kept dictionary/method for now.
There is a slight benefit for interface/attribute as we would return the same object every time, which might be handy if the metadata becomes a potentially big object.

[youennf]

One potential downsides of using a VideoFrameMetadata interface is that we would probably anyway need a VideoFrameMetadataInit dictionary.
Anytime a spec would extend VideoFrameMetadata with a new readonly attribute, it would also have to extend VideoFrameMetadataInit.

[youennf]

As a side note, VideoFrameMetadata name (even though dictionary names are not that important) conflicts with https://wicg.github.io/video-rvfc/#dictdef-videoframemetadata.
@tguilbert-google FYI.

[tguilbert-google]

I'd be fine with renaming the rVFC VideoFrameMetadata to something else, to avoid collision. I think that VideoFrameMetadata makes more sense here.

Albeit verbose, rVFC VideoFrameMetadata could be renamed to VideoFrameCallbackMetadata?

[youennf]

rVFC VideoFrameMetadata could be renamed to VideoFrameCallbackMetadata?

Sounds good, dictionary names are not exposed to JS so verbosity is not a big issue anyway.

[sandersdan]

Approach LGTM. The editors are more qualified to review the text of the changes.

[dalecurtis]

Thanks, lgtm. @aboba @padenot any objections?

[dalecurtis]

Still lgtm; thanks @youennf -- though there is some PR build error:

FATAL ERROR: Couldn't find 'webcodecs-video-frame-metadata-registry' in bibliography data.

[chrisn]

Should we have a registry entry to run the CfC or is it fine to keep it empty?

I think it's fine to be empty.

I'll raise another PR for the remaining procedural bits. We can probably also drop this item from the Media WG meeting agenda for tomorrow.

[chrisn]

@padenot, @aboba, is this OK to merge, and we can raise a separate PR for any remaining procedural requirements?

[padenot]

Yes.

[youennf]

@aboba, can you review again the PR and approve it if you think this is fine?

[dalecurtis]

Still lgtm; thanks @youennf -- though there is some PR build error:

FATAL ERROR: Couldn't find 'webcodecs-video-frame-metadata-registry' in bibliography data.

Bump on this one before submitting though.

[youennf]

@tidoust, can you shed some light on the issue @dalecurtis is mentioning above?

[tidoust]

@tidoust, can you shed some light on the issue @dalecurtis is mentioning above?

I pushed an update to add the entry to the local biblio. The entry can be removed when the registry document comes to existence and gets added to official biblio databases.

[dalecurtis]

Great, looks like we're all good here. Thanks @youennf!