Priorities for published and subscribed tracks

[lukasIO]

It'd be a very flexible approach to be able to manually set priorities for tracks. A (non-default) priority could influence the stream allocation based on the application needs by (de-)prioritizing specific tracks, making sure the bandwidth needs for the ones marked with higher priority are satisfied first.

It could be a hint for published tracks

const pub = await room.localParticipant.publishTrack(mediaStreamTrack, {
  name: 'mytrack',
  priority: 2
})

but in the best case scenario also for subscriptions

function handleTrackSubscribed(
  track: RemoteTrack,
  publication: RemoteTrackPublication,
  participant: RemoteParticipant
) {
  if (track.kind === Track.Kind.Video) {
    publication.setPriority(2)
  }
}

Common application examples I can think of:

• prioritizing all audio tracks over video tracks (or vice versa)
• prioritizing screen share tracks over regular video tracks
• prioritizing certain video (and audio) tracks of participants on a "stage"

I'm not sure if this would be in any way connected to the work going on around the stream allocator (e.g. #257).

[boks1971]

Thank you @lukasIO .

The priorities are something we are trying to do with sane defaults for now. But, we have also talked about explicit priority setting. Want to see if other signals we have is useful without explicit priority.

For example with the auto size management feature, the subscriber is subscribing to different quality based on visibility/size of the video element. That is one signal that drives priority.

Another is TrackSource (rationale being screen share (maybe for some exemption cases) should take higher priority).

You are correct. Explicit signals are very useful. But, we are checking how the other signals can inform this module and what other information will be needed from the clients.

[lukasIO]

Thanks for the explanation!

I completely see your arguments concerning sane defaults.
Where livekit (imho) shines is exactly this, a quick way to get started (sane defaults) without restricting the possibility to customize on a lower level the deeper you dive.

While I also see that TrackSource and on-screen-size are okish indicators, I don't think that they are sufficient. It's a sane assumption that a screentrack should have priority over other video tracks.
Staying in the same scenario of screensharing, I'd expect someone within the room to present something.
So there are a couple of things I might want to consider additionally:

• the webcam video of the person presenting (who might just be visible in a small thumbnail) still has priority over the others, who might have the same size on the screen, but are not as central to what's happening
• the audio of the presenter (both screen audio and mic tracks) need to be prioritized in order to not have a degraded quality influenced by video streams of other (not as central) participants

edit: this is just supposed be example in favor of manual prioritization...
As a general note about the prioritization, it would also be possible to set screentracks with a default higher priority on the clients, so the server would only have to worry about managing priorities and qualities and not specific types (like tracksource). Which would still allow the application developers to easily override any sane defaults

[boks1971]

Totally understood @lukasIO .

We are definitely thinking about adding some form of client control for priorities. But, like you mentioned we are trying to get a good balance of sane defaults and customisation.

The stream allocator is a tricky piece. So, we are working on it using sane defaults and learning about the challenges. That will educate us in what and how client side customisation should be designed.

[yogeshydvme]

This will surely help a lot!
+1