Introducing frame based stream tracker. #1267

boks1971 · 2022-12-28T04:33:39Z

Design is to have different implementations. Currently supported are

Packet based - the existing method
Frame based - new method, will require more experimentation to iron out rough edges. The idea (or goal or hope :-) ) behind this is to have something that can work automatically for different source types without any configuration. It is not there yet, but hopefully this is a step along the way.

Also made a few other changes

Do not remove layer if not simulcast
Do not do FPS calculation when paused

Will drop notes inline to explain some of the bits.

boks1971 · 2022-12-28T04:34:51Z

pkg/config/config.go

+	BitrateReportInterval map[int32]time.Duration             `yaml:"bitrate_report_interval,omitempty"`
+	ExemptedLayers        []int32                             `yaml:"exempted_layers,omitempty"`
+	PacketTracker         map[int32]StreamTrackerPacketConfig `yaml:"packet_tracker,omitempty"`
+	FrameTracker          map[int32]StreamTrackerFrameConfig  `yaml:"frame_tracker,omitempty"`


Reworking this to contain all the stream tracker config for a specific track source. Top level config then has only Video and Screenshare.

boks1971 · 2022-12-28T04:36:32Z

pkg/sfu/buffer/buffer.go

@@ -431,7 +439,7 @@ func (b *Buffer) patchExtPacket(ep *ExtPacket, buf []byte) *ExtPacket {
 }

 func (b *Buffer) doFpsCalc(ep *ExtPacket) {
-	if b.frameRateCalculated || len(ep.Packet.Payload) == 0 {
+	if b.paused || b.frameRateCalculated || len(ep.Packet.Payload) == 0 {


Passing along the paused state to Buffer. Main reason being, when camera is disabled on Firefox, looks like we cannot fully disable it. It sends 1 fps at layer 2 when disabled. That is causing the FPS calculator to calculate wrong frame rate for layer 2.

boks1971 · 2022-12-28T04:37:28Z

pkg/sfu/streamtracker/interfaces.go

+
+// ------------------------------------------------------------
+
+type StreamTrackerImpl interface {


The impl that different stream tracker algorithms should implement.

boks1971 · 2022-12-28T04:40:28Z

pkg/sfu/streamtracker/streamtracker.go


+type StreamTrackerParams struct {
+	StreamTrackerImpl     StreamTrackerImpl


This is the StreamTracker base struct that should be passed in an impl. This class does things like measuring bitrate, handling pause state, stream state etc.

I did another design where the actual implementation is the top level struct and those include this base struct so that callers could instantiate an implementation and be done with it. Now callers have to instantiate an impl and then put that in the params of base and instantiate base. Not too bad, but not ideal. But, the base embedded in top level became too complicate with locking and code sharing. So, went with this approach which turned out to be cleaner.

boks1971 · 2022-12-28T04:42:29Z

pkg/sfu/streamtracker/streamtracker_frame.go

+	Logger    logger.Logger
+}
+
+type StreamTrackerFrame struct {


The new frame based stream tracker.

boks1971 · 2022-12-28T04:46:48Z

pkg/sfu/streamtracker/streamtracker_frame.go

+func (s *StreamTrackerFrame) Observe(hasMarker bool, ts uint32) StreamStatusChange {
+	if !s.initialized {
+		s.initialized = true
+		if hasMarker {


I had this like

// Start only on a key frame. This condition should hold true as this path is exercised under following conditions // 1. When track starts // 2. When track resumes after an unmute. // 3. When a layer starts after dynacast pause. if isKeyFrame { s.initialized = true if hasMarker { s.tsInitialized = true s.oldestTS = ts s.newestTS = ts s.numFrames = 1 } return StreamStatusChangeActive }

But, it turned out to be not true. I thought browsers will use a key frame if the previous frame is too far back. But, they don't. If I disable camera, wait for 15 seconds and then re-start the camera, it sends a P frame with the old frame as reference. Makes sense to make a P frame if the old frame is in the encoder memory. So, removed that and just using re-start of traffic. This is effectively packet based, i. e. declaring active on first packet just like the packet based stream tracker works.

boks1971 · 2022-12-28T04:47:30Z

pkg/sfu/streamtracker/streamtracker_frame.go

+			// STREAM-TRACKER-FRAME-TODO: might need to protect against one frame, long pause and then one or more frames, i. e. window getting stale.
+			// One possible option is to reset the fps measurement variables (tsInitialized, oldestTS, newestTS, numFrames, lowestFrameRate, evelInterval)
+			// and restart the lowest frame rate calulation process.
+		}


Noting a couple of bits with this marker. I could not reproduce in my testing, but something to think about.

boks1971 · 2022-12-28T04:48:06Z

pkg/sfu/streamtracker/streamtracker_packet.go

@@ -0,0 +1,83 @@
+package streamtracker


No change in this algorithm, just split out into an impl.

boks1971 · 2022-12-28T04:48:58Z

pkg/sfu/streamtrackermanager.go

-	exemptedLayersVideo       []int32
-	exemptedLayersScreenshare []int32
+	trackerType   config.StreamTrackerType
+	trackerConfig config.StreamTrackerConfig


Reworking config made this simpler to just take Video or Screenshare based on track source and put it in this variable and use it.

boks1971 · 2022-12-28T04:50:58Z

pkg/sfu/streamtrackermanager.go

 	s.trackers[layer] = tracker
 	s.lock.Unlock()

+	tracker.SetPaused(paused)


paused also caused trouble with stream tracker. It was not passed in when a tracker was created. With FF, if the camera is disabled soon after start, layer 2 (the 1 FPS layer) starts up after pausing. That triggers OnTrack and a tracker is created. But, that tracker did not get the paused setting and proceeded to declare layer available seeing packets/frames from that layer.

boks1971 · 2022-12-28T04:51:39Z

pkg/sfu/streamtrackermanager.go

@@ -414,7 +446,8 @@ func (s *StreamTrackerManager) removeAvailableLayer(layer int32) {

 	newLayers := make([]int32, 0, DefaultMaxLayerSpatial+1)
 	for _, l := range s.availableLayers {
-		if exempt || l != layer {
+		// do not remove layers for non-simulcast
+		if exempt || l != layer || len(s.trackInfo.Layers) < 2 {


Making this change to not remove layer for non-simulcast tracks.

boks1971 · 2022-12-28T06:09:50Z

pkg/sfu/streamtrackermanager.go

+		s.trackerConfig = trackersConfig.Screenshare
+	case livekit.TrackSource_CAMERA:
+		s.trackerConfig = trackersConfig.Video
+	default:


Using Video settings by default as some of the tests don't have TrackSource.

cnderrauber

Looks good!

davidzhao

lg!

davidzhao · 2022-12-28T07:17:41Z

pkg/sfu/streamtracker/interfaces.go

+
+// ------------------------------------------------------------
+
+type StreamTrackerImpl interface {


boks1971 added 8 commits December 26, 2022 21:36

Split stream tracker impl from base

8586f28

slight re-arrangement of code

c7a8b61

Merge remote-tracking branch 'origin/master' into raja_stream_tracker

7ac345a

Merge remote-tracking branch 'origin/master' into raja_stream_tracker

8a08339

Merge remote-tracking branch 'origin/master' into raja_stream_tracker

8caadb7

fps based stream tracker

890f844

MinFPS config

b11a432

switch back to packet based tracker

68c466d

boks1971 commented Dec 28, 2022

View reviewed changes

boks1971 requested review from cnderrauber and davidzhao December 28, 2022 04:54

use video config by default to handle sources without type

16d1099

boks1971 commented Dec 28, 2022

View reviewed changes

cnderrauber approved these changes Dec 28, 2022

View reviewed changes

davidzhao approved these changes Dec 28, 2022

View reviewed changes

boks1971 merged commit 2b031a5 into master Dec 28, 2022

boks1971 deleted the raja_stream_tracker branch December 28, 2022 07:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introducing frame based stream tracker. #1267

Introducing frame based stream tracker. #1267

boks1971 commented Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022 •

edited

boks1971 Dec 28, 2022

davidzhao Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

boks1971 Dec 28, 2022

cnderrauber left a comment

davidzhao left a comment

davidzhao Dec 28, 2022


		// ------------------------------------------------------------

		type StreamTrackerImpl interface {


		type StreamTrackerParams struct {
		StreamTrackerImpl StreamTrackerImpl

Introducing frame based stream tracker. #1267

Introducing frame based stream tracker. #1267

Conversation

boks1971 commented Dec 28, 2022

Choose a reason for hiding this comment

boks1971 Dec 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cnderrauber left a comment

Choose a reason for hiding this comment

davidzhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boks1971 Dec 28, 2022 •

edited