Legacy getRemoteStreams() in Unified Plan

[henbos]

Chrome's legacy getRemoteStreams() is currently, under Plan B, defined as:

All streams of all receivers.

This works because inactive receivers are removed. In Unified Plan, this does not work because transceivers stick around forever. We can look at their currentDirection though:

All streams of all active receivers. An active receiver is a receiver whose transceiver's currentDirection is "sendrecv" or "recvonly".

currentDirection is chosen, as opposed to direction, because otherwise merely offering to receive would count as a receiver being "active". This does not jive with current definitions of getRemoteStreams().

There is a problem here though. The receiver's currentDirection is not updated at setRemoteDescription(offer), but rather at setLocalDescription(answer). We get a changing behavior:

const offer = await caller.createOffer();
await caller.setLocalDescription(offer);
// In Plan B, this updates callee.getRemoteStreams():
await callee.setRemoteDescription(offer);
const answer = await callee.createAnswer();
// In Unified Plan, this is where callee.getRemoteStreams() is updated:
await callee.setLocalDescription(answer);
await caller.setRemoteDescription(answer);

In order for the browser or application to know what receivers are "active" we need to expose what direction was offered/answered by the remote end.

Questions:

1. Is this what [[FiredDirection]] is? Added in Fire events in SLD(answer) in direction rejection edge-case. #1867.
2. Does this motivate exposing this as an attribute to the application?

[henbos]

On second thought, a receiver that is not "active" yet will not have any remote streams associated with it, so maybe the original "All streams of all receivers" will work anyway.

[henbos]

I'm leaving the issue open for now in case there is something worth discussing, e.g. is the receiver's set of associated streams guaranteed to reflect the notion of "getRemoteStreams()", but feel free to close this issue if I'm not mistaken.

[alvestrand]

getRemoteStreams isn't just legacy, it's been removed from the spec.
So formally, implementations are free to implement it (or not) as they see fit.

Also leaving the issue open in case there are comments needed, but (strictly with my formal WG hat on) intending to close it after a while as "none of our business"

[henbos]

getRemoteStreams() is not in the spec but we should zoom out and look at the use case it highlights, and discuss whether or not there is a legitimate need to know which are the "active" receivers.

1. const offer = await caller.createOffer();
2. await caller.setLocalDescription(offer);
3. await callee.setRemoteDescription(offer);
4. const answer = await callee.createAnswer();
5. await callee.setLocalDescription(answer);
6. await caller.setRemoteDescription(answer);

"What are the active receivers?" was something the application used to be able to easily figure out at step 3 upon the promise resolving (in Chrome).
This question is now only easily answerable at step 5, half-way through the answer cycle, by looking at currentDirection.

With correct ontrack and onmute wiring the application can figure out which receivers are currently "active" (unless remote onmute can fire for other reasons?), but it's not as easy as "getReceivers() and look at direction/currentDirection".

So before we close this - from a usability and shimmability standpoint: Do we care?

[henbos]

It may be possible for the application to maintain its own set of "active receivers", that it adds to from ontrack and removes from on onmute, but it is not as convenient as "getRemoteStreams()", and getReceivers()/RTCRtpReceiver does not seem feature complete if there is no way to tell whether or not it is "active".

[alvestrand]

A good question wrt "what are the active receivers" is "why do you want to know?".

The answers to

pc.getReceivers().filter(r => { if (!r.stream.ended) return r; })
pc.getReceivers().filter(r => { if (!r.stream.muted) return r; })
pc.getReceivers().filter(r => { if (r.getStreams()) return r; })

should all be able to be different, and only the last one seems like it's relevant to getRemoteStreams().

I suggest removing the discussion of getRemoteStreams() from the question of active receivers; they seem orthogonal to me.

[alvestrand]

(note: I think we should have a getStreams() method on RTCRTPReceiver, which returns the streams that have been created based on the stream IDs that the receiver has been signalled in this media section. There is currently no way to get at the information about the stream IDs that have been signalled, and the mapping from receiver to stream can't be performed by just looking at the IDs (I think), so accessing this internal state needs a new API.)

[henbos]

Currently, the receiver's streams is an internal slot, but it is exposed in the ontrack event. I agree this information should be exposed, else we force people to listen to ontrack & friends to maintain their own version of this "slot".
But even with r.getStreams(), you might not have any streams though (streamless track) so you can't rely on streams' states.

Because transceivers are reusable r.track.ended will be false even for inactive ones (ended is a state you can't get out of).
What we do have is r.track.muted, but I don't think this works either, unless I'm mistaken they are supposed to be muted by default and become unmuted when frames arrive? Is that true?
Maybe r.track.readyState == 'live', or is this the same as unmuted?

I would argue that if you give me a PC I should be able to inspect its state without having had to listen to a history of events to piece together what its state are from ontrack & friends. This would be useful for any utils.

[henbos]

I'll file a new issue

[henbos]

I split this up into #1920 and #1921, closing this one.