Life-time management of VideoFrame when used with streams

[youennf]

VideoFrame objects require to call close so as to clean resources in advance of garbage collection.
This is important as VideoFrame may be big and scarce resources.
Streams currently rely on garbage collection to do the lifetime management, which is an important mismatch with VideoFrame model.

The first consequence is that it is unclear what the API contract is when dealing with streams of VideoFrame.
When reading a ReadableStream, should it be the source or the reader that call close.
Ditto for WritableStream.

It would be good to get a clear understanding of what is desired, document the ideal API contract and, if possible, enforce part or all of the contract in the API and algorithms.

The second issue is that, if a stream has VideoFramed being queued, and the stream gets aborted, we want the VideoFrames to be closed, or at least leave the opportunity for the web application to do that clean-up.

[dalecurtis]

https://github.com/whatwg/streams/blob/main/streams-for-raw-video-explainer.md#transferring-ownership-streams-explained was a proposal for addressing this. Did anything more come from these conversations? This was highlighted by @tidoust as one of the more painful issues he encountered: https://lists.w3.org/Archives/Public/public-media-wg/2023Mar/0010.html

[youennf]

I think the plan is pretty clear (as described in the raw video explainer).
It is now mostly a matter of prioritizing the editing and implementation work.
I can probably help with the editing work.

[dalecurtis]

Okay, I wasn't sure if we had consensus on the direction, but it seems reasonable to me. @ricea @domenic in case they have opinions.

[ricea]

I think the plan is pretty clear (as described in the raw video explainer). It is now mostly a matter of prioritizing the editing and implementation work. I can probably help with the editing work.

I'm not certain whether we want a new kind of controller or a flag on the existing DefaultController that enables transfers. However, if you could draft something then that would be a big help in seeing what approach is simpler.

[youennf]

I was thinking of just adding a new flag on default controllers.
I did an initial pass (ReadableStream only for now) at #1271.
This approach looks ok to me. Thoughts?

[ricea]

I made some comments over there but in general it looks good.