Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add vp9 svc support by Dependency Descriptor #1586

Merged
merged 3 commits into from
Apr 6, 2023
Merged

Add vp9 svc support by Dependency Descriptor #1586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2e908c2
Add VP9 SVC support
cnderrauber Apr 6, 2023
2036af4
Fix preferred fps does not work
cnderrauber Apr 6, 2023
e8e1e09
Fix forwarder test
cnderrauber Apr 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 8 additions & 10 deletions pkg/rtc/mediaengine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,14 @@ func registerCodecs(me *webrtc.MediaEngine, codecs []*livekit.Codec, rtcpFeedbac
RTPCodecCapability: webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeVP8, ClockRate: 90000, RTCPFeedback: rtcpFeedback.Video},
PayloadType: 96,
},
/*
{
RTPCodecCapability: webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeVP9, ClockRate: 90000, SDPFmtpLine: "profile-id=0", RTCPFeedback: rtcpFeedback.Video},
PayloadType: 98,
},
{
RTPCodecCapability: webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeVP9, ClockRate: 90000, SDPFmtpLine: "profile-id=1", RTCPFeedback: rtcpFeedback.Video},
PayloadType: 100,
},
*/
{
RTPCodecCapability: webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeVP9, ClockRate: 90000, SDPFmtpLine: "profile-id=0", RTCPFeedback: rtcpFeedback.Video},
PayloadType: 98,
},
{
RTPCodecCapability: webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeVP9, ClockRate: 90000, SDPFmtpLine: "profile-id=1", RTCPFeedback: rtcpFeedback.Video},
PayloadType: 100,
},
{
RTPCodecCapability: webrtc.RTPCodecCapability{MimeType: webrtc.MimeTypeH264, ClockRate: 90000, SDPFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f", RTCPFeedback: rtcpFeedback.Video},
PayloadType: 125,
Expand Down
2 changes: 1 addition & 1 deletion pkg/rtc/participant_sdp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ func (p *ParticipantImpl) setCodecPreferencesVideoForPublisher(offer webrtc.Sess

mime = strings.ToUpper(mime)
// remove dd extension if av1 not preferred
if !strings.Contains(mime, "AV1") {
if !strings.Contains(mime, "AV1") && !strings.Contains(mime, "VP9") {
for i, attr := range unmatchVideo.Attributes {
if strings.Contains(attr.Value, dd.ExtensionUrl) {
unmatchVideo.Attributes[i] = unmatchVideo.Attributes[len(unmatchVideo.Attributes)-1]
Expand Down
2 changes: 2 additions & 0 deletions pkg/sfu/buffer/buffer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,8 @@ func (b *Buffer) getExtPacket(rtpPacket *rtp.Packet, arrivalTime int64) *ExtPack
ep.KeyFrame = IsH264Keyframe(rtpPacket.Payload)
case "video/av1":
ep.KeyFrame = IsAV1Keyframe(rtpPacket.Payload)
case "video/vp9":
ep.KeyFrame = IsVP9Keyframe(rtpPacket.Payload)
}

if ep.KeyFrame {
Expand Down
26 changes: 26 additions & 0 deletions pkg/sfu/buffer/helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import (
"encoding/binary"
"errors"

"github.com/pion/rtp/codecs"

"github.com/livekit/protocol/logger"
)

Expand Down Expand Up @@ -351,4 +353,28 @@ func IsAV1Keyframe(payload []byte) bool {
}
}

// IsVP9Keyframe detects if vp9 payload is a keyframe
// taken from https://github.com/jech/galene/blob/master/codecs/codecs.go
// all credits belongs to Juliusz Chroboczek @jech and the awesome Galene SFU
func IsVP9Keyframe(payload []byte) bool {
var vp9 codecs.VP9Packet
_, err := vp9.Unmarshal(payload)
if err != nil || len(vp9.Payload) < 1 {
return false
}
if !vp9.B {
return false
}

if (vp9.Payload[0] & 0xc0) != 0x80 {
return false
}

profile := (vp9.Payload[0] >> 4) & 0x3
if profile != 3 {
return (vp9.Payload[0] & 0xC) == 0
}
return (vp9.Payload[0] & 0x6) == 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't fully understand this. I cannot read the bit stream format document :-). But, this RFC seems to say that we do not need to dig beyond the VP9 Payload Descriptor to determine key frame. I may change this function based on that when I do my refactor. Guess, we really do not need to know if key frame or not with dependency descriptor. We will need that only when using simulcast.

https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16. The actual text

      A key picture is a picture whose base
      spatial layer frame is a key frame, and which thus completely
      resets the encoder state.  This packet will have its P bit equal
      to zero, SID or L bit (described below) equal to zero, and B bit
      (described below) equal to 1.

This is what I used in the refactor branch I am working on. I will compare with the above and check if they match.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guess, we really do not need to know if key frame or not with dependency descriptor.

That's true, actually, it works well without this keyframe detection function, just add it to be consistent with other codecs.

}

// -------------------------------------
10 changes: 5 additions & 5 deletions pkg/sfu/forwarder.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,8 +273,8 @@ func (f *Forwarder) DetermineCodec(codec webrtc.RTPCodecCapability) {
case "video/vp8":
f.isTemporalSupported = true
f.vp8Munger = NewVP8Munger(f.logger)
case "video/av1":
// TODO : we only enable dd layer selector for av1 now, at future we can
case "video/av1", "video/vp9":
// TODO : we only enable dd layer selector for av1 and vp9 now, at future we can
// enable it for vp8 too
f.ddLayerSelector = NewDDVideoLayerSelector(f.logger)
}
Expand Down Expand Up @@ -515,7 +515,7 @@ func (f *Forwarder) AllocateOptimal(availableLayers []int32, brs Bitrates, allow
}
alloc.TargetLayers = buffer.VideoLayer{
Spatial: int32(math.Min(float64(f.maxPublishedLayer), float64(maxSpatial))),
Temporal: buffer.DefaultMaxLayerTemporal,
Temporal: f.maxLayers.Temporal,
}
}

Expand Down Expand Up @@ -570,14 +570,14 @@ func (f *Forwarder) AllocateOptimal(availableLayers []int32, brs Bitrates, allow
alloc.TargetLayers.Spatial = l
}
}
alloc.TargetLayers.Temporal = buffer.DefaultMaxLayerTemporal
alloc.TargetLayers.Temporal = f.maxLayers.Temporal
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change to fix client preferred fps (temporary layer), can you verify it is correct? @boks1971

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be okay @cnderrauber , but we probably need to allow that opportunistic higher layer. I will looking at adding it back for simulcast tracks and use proper layer for DD tracks.


alloc.RequestLayerSpatial = alloc.TargetLayers.Spatial
} else {
requestLayerSpatial := int32(math.Min(float64(f.maxLayers.Spatial), float64(f.maxPublishedLayer)))
if f.currentLayers.IsValid() && requestLayerSpatial == f.requestLayerSpatial && f.currentLayers.Spatial == f.requestLayerSpatial {
// current is locked to desired, stay there
alloc.TargetLayers = f.currentLayers
alloc.TargetLayers = buffer.VideoLayer{Spatial: f.requestLayerSpatial, Temporal: f.maxLayers.Temporal}
alloc.RequestLayerSpatial = f.requestLayerSpatial
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might have to revert these changes back to how they were and deal with layer limiting based on DD or not. But, I will take care of it in the refactor branch.

} else {
// opportunistically latch on to anything
Expand Down
4 changes: 2 additions & 2 deletions pkg/sfu/forwarder_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,7 +387,7 @@ func TestForwarderAllocateOptimal(t *testing.T) {
f.requestLayerSpatial = 0
expectedTargetLayers = buffer.VideoLayer{
Spatial: 2,
Temporal: 3,
Temporal: 1,
}
expectedResult = VideoAllocation{
PauseReason: VideoPauseReasonFeedDry,
Expand All @@ -397,7 +397,7 @@ func TestForwarderAllocateOptimal(t *testing.T) {
TargetLayers: expectedTargetLayers,
RequestLayerSpatial: 2,
MaxLayers: f.maxLayers,
DistanceToDesired: -1.5,
DistanceToDesired: -1,
}
result = f.AllocateOptimal([]int32{0, 1}, emptyBitrates, true)
require.Equal(t, expectedResult, result)
Expand Down