VideoFrame metadata

[chcunningham]

Some uses of VideoFrame would like to associate named metadata with the frame. Examples include

• ImageDecoder (not yet part of the spec, but see Chromium prototype here) could use frame.metadata[complete] = true/false to distinguish partial vs complete frames for progressively downloaded images.
• FaceMesh (separate proposal) data

Alternative designs should also be considered, including subclassing or wrapping VideoFrame to add the metadata.

[chcunningham]

@sandersdan - I think there were other proposals to include user-provided metadata? Is there a user request behind that? Seems tricky to always map input:output.

[sandersdan]

There was also the RTCEncodedVideoFrame feature gap (internal e-mail thread "RTCEncodedVideoFrame : WebCodecs"). I'm not sure if I've seen a user request, but I expect many cases where some format metadata or network timing information would be convenient to pass that way.

I think at this point I'm resigned to mapping input packets to output frames 1:1, with the caveat that some might get dropped or duplicated in corner cases.

Bikeshed: if we add a metadata object, we should recommend that specs use Symbol rather than String keys so that they can't conflict with app code.

[chcunningham]

The ImageDecoder 'complete' flag was solved separately using ImageDecodeResult, which has a complete flag.
The FaceMesh idea is not actively being pursued.

But the need for metadata lives on in issue #189. I'll close this in favor of that and paste some bits of the discussion from here into that bug.