Make MediaStreamTrack serializable

[jan-ivar]

@annevk says most objects that are transferable are also serializable.

MediaStreamTrack has a custom clone() method today, like VideoFrame has. But VideoFrame is serializable while MediaStreamTrack is not.

After looking over our custom clone algorithm in w3c/mediacapture-main#821, it seems semantically compatible to me, and since tracks are already transferable, I think it would make sense to make them serializable as well. This should simplify our algorithms, and make the following work:

const stream = await navigator.mediaDevices.getUserMedia({video: true});
const [track] = stream.getVideoTracks();
const clone = structuredClone(track);

Today this produces DataCloneError: The object could not be cloned. in Firefox Nightly.

[eladalon1983]

I understand what it means to serialize a VideoFrame, write it to disk, read it later. A frame seems to me equally useful on the machine where it was first produced, as on a machine where it is recreated. What, though, is the meaning of deserializing a track on another machine, that has different peripherals attached and different permissions set?

[annevk]

Note that you don't necessarily have to support storage (that's what the forStorage argument is for). And at least thus far storage is coupled to the same device, though I suppose a user agent could offer to synchronize it.

[youennf]

I think we discussed this during a past WG meeting.
IIRC, the room consensus was that transferable was good enough for now.
As of structuredClone, I would guess structuredClone(track, [track]); would work just fine.

[jan-ivar]

Right, so since we can shim structuredClone(track) as:

structuredClone(track, {transfer: [track.clone()]});

...my question is why not support it outright?

[jan-ivar]

I understand what it means to serialize a VideoFrame, write it to disk, read it later.

VideoFrame does not support forStorage. It's a ref-counted handle to (potentially GPU backed) buffers owned by the UA.

[youennf]

I think we should look at the meeting minutes, my recollection is that tracks are expensive objects, they have a potential impact on privacy indicators, temporary permissions...
Also, contrary to VideoFrame, a live track with event listeners might not be easy to GC.
Having an explicit cloning step allows keeping the number of live tracks low.

...my question is why not support it outright?

My question is why supporting it outright?

[jan-ivar]

@youennf Privacy indicators and temporary permissions are tied to the context that created it.

The track objects themselves are not expensive, as the cloning steps reveal. They are mere (ref-counted) handles to the singular underlying source.

Also, contrary to VideoFrame, a live track with event listeners might not be easy to GC.

If the concern is accidental clones, then event handlers aren't an issue. JS adding event handlers registers an interest. The JS is on the hook to call track.stop() when it is finished with a track. That's already the contract.

Having an explicit cloning step allows keeping the number of live tracks low.

How so? Why is structuredClone(track) any different from track.clone() in this regard?

I agree it can be a hassle for apps to keep track of clones, but I don't see this moving the needle one way or the other.

My question is why supporting it outright?

Simpler spec and model. Developers can already do stream.clone() and get a cloned stream containing clones of all tracks. This should fall out naturally with structuredClone, which would support other collections as well.

[alvestrand]

Since I argued earlier for defining serialize and defining transfer in terms of serialize + stop, I can't say that I see much bad happening because of this :-)

I do think we should have only one algorithm spelled out in the spec and define the other in terms of the spelled-out one.