When are keyframes generated?

[aboba]

Keyframe generation in WebRTC-PC is an under-specified side effect of other operations.

• Key frames may be generated when changing scaleResolutionDownBy and re-setting active=true after active=false. As noted below, this may not be necessary if the receiver supports resolution scaling.
• Keyframes typically aren't generated when setting active = false (e.g. turning off the highest simulcast layer) or dropping the highest SVC spatial layer. This causes SFU to switch participants to a lower layer, but without a keyframe or layer refresh they won’t be able to decode it.

The current text relating to scaleResolutionDownBy and active does not provide much detail about what happens when a value of scaleResolutionDownBy is present in setParameters() or when active is changed. For example:

• Is a keyframe automatically generated when the scaleResolutionDownBy value is changed or when active is set to true? Can a keyframe be automatically generated in other situations (e.g. when scaleResolutionDownBy is set, regardless of whether it has changed, when active is set to false in a simulcast layer)?
• If a keyframe is not automatically generated, what happens if the decoder doesn't support the ability to do spatial prediction, that is, using frames of a resolution different than the current resolution as dependencies? In this circumstance a change in the scaleResolutionDownBy value can be expected to result in a decoder error and a PLI.

Note: the Media Capabilities API makes it possible to discover whether a decoder supports spatial prediction via the spatialScalability attribute. This was introduced because some (hardware) decoders were found to be unable to do spatial prediction, and therefore would produce a decoder error on encountering an encoding using spatial prediction.

[dontcallmedom-bot]

This issue was discussed in WebRTC May 2023 meeting – 16 May 2023 (Issue #158 / PR #167: Requesting a key frame via setParameters)