Bandwidth allocation is the process of selecting the set of layers to forward to a specific endpoint (the "receiver"), or "allocating" the available bandwidth among the available layers. When conditions change, the algorithm is re-run, and the set of forwarded layers is updated according to the new result.
The overall goal is to provide the most relevant and suitable set of streams to the receiver, given a limited bandwidth.
The available bandwidth (or the Bandwidth Estimation, BWE) is the estimated bandwidth between the bridge and the receiver. It is calculated elsewhere, and is only used as input for bandwidth allocation.
This is the list of streams being sent from the other endpoints in the conference. Streams have multiple layers, and the algorithm selects one layer (or no layers) for each source.
For example, a simulcast sender can encode its video in 3 different encodings, with 3 different frame rates each. This gives the allocator 9 layers to choose from.
The list of available sources changes when endpoints join or leave the conference, or when they signal a change in their streams (such as when an endpoint switches from a video camera to screensharing or vice-versa).
The following settings are controlled by the receiver, with messages over the "bridge channel".
LastN is the maximum number of video streams that the receiver wants to receive. To effectively stop receiving video (for example to conserve bandwidth), the receiver can set LastN=0.
This is a list of sources to be prioritized first, overriding the natural speech activity order of the endpoints which own the sources.
For example, if the receiver wants to always receive a screensharing source, regardless of who is speaking in the conference, it can "select" this source.
This is a list of sources for which allocation should be prioritized up to a higher resolution (since they are displayed "on stage"). On-stage sources are prioritized higher than selected sources, and in addition:
- Allocation for them is greedy up to the preferred resolution (360p by default)
- Above the preferred resolution, only frame rates of at least 30 fps are considered.
Video constraints are resolution (maxHeight) and frame rate (maxFrameRate) constraints for each source. These are
"soft" constraints in the sense that the bridge may exceed them in some circumstances (see below).
When set to a negative number, they indicate no constraints.
When set to 0, they signal that no video should be forwarded for the associated source, and this is never exceeded.
When set to a positive number, the algorithm will attempt to select a layer which satisfies the constraints. If no layers satisfy the constraints, and there is sufficient bandwidth, the algorithm will exceed the constraints and select the lowest layer. In practice this is relevant only in the case where a sender does not use simulcast (or SVC), and encodes a single high-resolution stream. Given enough bandwidth, the stream will be forwarded even when the receiver signaled low constraints.
The bandwidth allocation algorithm is implemented in BandwidthAllocator
It consists of 3 phases:
This phase orders the available sources in the desired way. It starts with sources coming from the endpoints ordered by speech activity (dominant speaker, followed by the previous dominant speaker, etc). Then, it moves the sources from the endpoints which are NOT sending video to the bottom of the list (this is actually implemented in ConferenceSpeechActivity. Finally, the selected sources are moved to the TOP of the list.
TODO: Update the algorithm, to only move selected endpoint when they are sending video.
This phase disables the video sources in the list that are not among the first LastN. Note that the effective
LastN value comes from the number signaled by the client, potentially also limited by static
and dynamic
configuration of the bridge. This is implemented by setting the maxHeight constraint to 0.
The resulting constraints are the "effective" constraints used by the rest of the algorithm. Once calculated, they are announced via an event, so that the sender-side constraints can be applied. Doing this step here, early in the process, allows us to do "aggressive layer suspension" (i.e. set sender-side constraints based on LastN).
The final phase is the actual allocation.
The first step is to initialize a list of layers to consider for each source. It starts with the list of all layers for the source, and prunes ones which should not be considered:
A) The ones with resolution and frame rate higher than the constraints
B) The ones which are inactive (the sending endpoint is currently not transmitting them)
C) Layers with high resolution but insufficient frame rate, that is at least the preferred resolution, and frame rate less than the preferred frame rate. For example, with the defaults of preferred resolution 360p and preferred frame rate 30 fps, the following layers will not be considered: 360p/7.5fps, 360p/15fps, 720p/7.5fps, 720p/15fps.
It starts with no layers selected for any source, and remaining bandwidth equal to the total available bandwidth.
Until there is remaining bandwidth, it loops over the sources in the order obtained in phase 1,
and tries to improve() the layer of each.
The normal improve() step selects the next higher layer if there is sufficient bandwidth. For on-stage sources
the improve() step works eagerly up to the "preferred" resolution.
The preferred resolution can be configured.
This section describes the signaling between the client and the bridge that affects bandwidth allocation.
The receiver's video constraint message is used to signal the preference of the client in regard to which media streams it wants to receive. Usually only a portion of all available videos is displayed on the client. Each video is identified by a source name and each endpoint can send multiple videos. The default format used in jitsi-meet follows a pattern where the first part is the endpoint ID followed by '-v' and the zero-based index of the video source (see the example below).
{
"colibriClass": "ReceiverVideoConstraints",
"lastN": 2,
"selectedSources": ["A-v0", "B-v0"],
"onStageSources": ["A-v1","C-v0", "D-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v1": { "maxHeight": 720 },
"B-v0": { "maxHeight": 360 }
}
}All fields are optional. The ones which are included will be updated, and the ones which are not included are not changed.
The defaultConstraints are used for sources not explicitly included in constraints (including new sources).
The initial values are lastN: -1 (unlimited), defaultConstraints: {maxHeight: 180}
(configurable),
and the rest empty.
Stage view with source A-v0 in high definition and all other sources in 180p:
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 }
}
}Stage view with source A-v0 in high definition, B-v0, C-v0, D-v0 in 180p and all others disabled:
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0"],
"defaultConstraints": { "maxHeight": 0 },
"constraints": {
"A-v0": { "maxHeight": 720 },
"B-v0": { "maxHeight": 180 },
"C-v0": { "maxHeight": 180 },
"D-v0": { "maxHeight": 180 }
}
}Stage view with source A-v0 in high definition, B-v0, C-v0, D-v0 disabled and all others in 180p:
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 },
"B-v0": { "maxHeight": 0 },
"C-v0": { "maxHeight": 0 },
"D-v0": { "maxHeight": 0 }
}
}Stage view with source A-v0 in high definition and all other sources in 180p, with "D-v0" prioritized higher than
the dominant speaker's video source:
{
"colibriClass": "ReceiverVideoConstraints",
"onStageEndpoints": ["A-v0"],
"selectedEndpoints": ["D-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 }
}
}Tile view with all sources in 180p/15fps:
{
"colibriClass": "ReceiverVideoConstraints",
"defaultConstraints": { "maxHeight": 180, "maxFrameRate": 15 }
}Tile view with all sources in 360p:
{
"colibriClass": "ReceiverVideoConstraints",
"defaultConstraints": { "maxHeight": 360 }
}Tile view with 180p, sources A-v0 and B-v0 prioritized, and sources C-v0 and D-v0 disabled:
{
"colibriClass": "ReceiverVideoConstraints",
"selectedSources": ["A-v0", "B-v0"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"C-v0": { "maxHeight": 0 },
"D-v0": { "maxHeight": 0 }
}
}Tile view with all sources disabled except A-v0, B-v0, C-v0:
{
"colibriClass": "ReceiverVideoConstraints",
"defaultConstraints": { "maxHeight": 0 },
"constraints": {
"A-v0": { "maxHeight": 180 },
"B-v0": { "maxHeight": 180 },
"C-v0": { "maxHeight": 180 }
}
}With two on-stage sources, and up-to 4 other sources at 180p:
{
"colibriClass": "ReceiverVideoConstraints",
"onStageSources": ["A-v0", "B-v0"],
"lastN": 6,
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A-v0": { "maxHeight": 720 },
"B-v0": { "maxHeight": 720 }
}
}The old format works with endpoint IDs rather than source names. It also uses selectedEndpoints and onStageEndpoints
instead of selectedSources and onStageSources. All use cases described in the examples above are valid here as
well, but the assumption is that every endpoint has only one video source.
{
"colibriClass": "ReceiverVideoConstraints",
"lastN": 2,
"selectedEndpoints": ["A", "B"],
"onStageEndpoints": ["C", "D"],
"defaultConstraints": { "maxHeight": 180 },
"constraints": {
"A": { "maxHeight": 720 },
"B": { "maxHeight": 360 }
}
}The support for source names vs legacy endpoint ID based format is determined on the ColibriV2 level. The endpoint create request must indicate "source-names" capability on the endpoint that will be using the new format. This is done automatically by Jicofo when multi stream support is enabled in jitsi-meet, but if you're using JVB without Jicofo then remember to do that yourself.
<iq xmlns='jabber:client' to='jvbbrewery@internal-muc.meet.jitsi/jvb1' id='VFLJ9-10' type='get'>
<conference-modify xmlns='jitsi:colibri2' meeting-id='62a5bc4c-c79c-4eab-a071-6740eb549296'>
<endpoint xmlns='jitsi:colibri2' id='fefbee3e' create='true' >
<capability name='source-names'/> <!--- SOURCE NAME CAPABILITY -->
<media type='audio'>
...
</media>
<media type='video'>
...
</media>
<transport ice-controlling='true'/>
</endpoint>
</conference-modify>
</iq>