JSMPEG Fallback

[lngarrett]

I have an issue where in grid mode, my cameras will start in MSE, but over several minutes they fallback to JSMPEG. Nothing obvious is jumping out to me in console or server logs. I'd like to self troubleshoot this a bit. So, my questions currently is if there's any guidance on better observing what might be happening here.

Looking at the code it appears that the key fallback mechanism occurs when:

• MsePlayer encounters errors (stalled buffer, decode issues, or startup problems)
• After 3+ errors, it calls onError?.("startup") to signal LivePlayer
• LivePlayer then resets and potentially changes player type

Are these areas I should light up with more logging or are there other areas I should be focused?

FWIW, this issue has been present even in .15. I have solid network connectivity and strong hardware on both client and server. I am running 15 cameras, I have a hunch that might have something to do with it. I did switch to substreams in .16, but that hasn't helped.

cameras:
  westcam:
    enabled: true
    ffmpeg:
      inputs:
        # Sub-stream
        - path:
            rtsp://foo:bar@foobar:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - detect
        # Main-stream
        - path:
            rtsp://foo:bar@foobar:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
    live: # .16
      streams:
        Main Stream: westcam
        Sub Stream: westcam_sub
    detect:
      enabled: true
      width: 1280
      height: 720
    objects:
      track:
        - person
        - car
    motion:
      mask: 0.797,0.927,0.925,0.924,0.929,0.977,0.795,0.981
    zones:
      Near_House:
        coordinates: 0.002,0.992,0,0,0.756,0,0.943,0.525,0.972,0.996
        loitering_time: 0
        objects: person
        inertia: 3
    review:
      alerts:
        required_zones: Near_House

go2rtc:
  rtsp:
    username: foo
    password: bar
  streams:
    parkingcam:
      - rtsp://foo:bar@foobar:554/cam/realmonitor?channel=1&subtype=0
    parkingcam_sub:
      - ffmpeg:rtsp://foo:bar@foobar:554/cam/realmonitor?channel=1&subtype=1#video=h264#width=iw*1.205#height=ih#audio=copy

[hawkeye217]

Your assessment of the code is indeed how it works. The high level explanation is documented in the Live docs in dev, specifically in the FAQ: https://deploy-preview-16390--frigate-docs.netlify.app/configuration/live#live-view-faq

With a fair number of cameras, your browser's decoder has to juggle loading several streams at once. Frigate is "eager" to fall back to jsmpeg because we intentionally prioritize showing an actual up-to-date live view of your camera's stream as quickly as possible.

[hawkeye217]

Has there been any thought around encoding the grid server side and sending a single stream?

Frigate already does this with Birdseye.

I mention this because I have a hunch that there's just a lot of overhead in opening so many individual streams.

Yeah, that's likely the case. I see a fallback fairly often if more than a few of my cams are trying to load their streams at once. But when using smart streaming, I wrote a reset into the player mode when activity stops.

[lngarrett]

Yeah, sorry I should've mentioned I am familiar with birdseye. I was alluding more towards making it as interactive as the grid view is, supporting higher frame rates, dynamic layouts etc. A big change to be sure. I'll hack on the grid view this weekend and see if I can work out patterns in the fallback that could be adjusted.

FWIW, I am using continuous sub stream on most of my more important cameras - those are likely the ones I'm seeing this issue with.

[hawkeye217]

I was alluding more towards making it as interactive as the grid view is, supporting higher frame rates, dynamic layouts etc. A big change to be sure.

That would require decoding the high-res streams, which would dramatically increase CPU usage.

FWIW, I am using continuous sub stream on most of my more important cameras - those are likely the ones I'm seeing this issue with.

That would make sense. Is there a particular reason beyond preference why the smart streaming option is not feasible for you? Since a static snapshot of your stream looks the same as a stream with no motion/objects/activity, smart streaming offers a tremendous bandwidth and resource savings.

[lngarrett]

Yes, the reason is that I've caught it not working a few times. Unfortunately, that sort of kills trust in the method. I'll see a car drive by in one camera, and on a camera with overlapping view - nothing. It's not always, but it's just hard to trust once it's proven to miss things. There's a certain sense of positive signal one gets from seeing timestamps ticking. Or in a less severe failure, it will lag a few seconds which can be enough to miss half the action.

Otherwise, I do agree with the principle. But, the challenge is that a camera with no motion and a camera that's failed to switch live also look the same.

[hawkeye217]

The main reason why a failure to switch would happen would be because your motion detection is not sensitive enough. Since you've seen the code, you know that any motion whatsoever will cause the player to engage.

What are your threshold and contour_area settings for your cameras? The Motion Tuner in Settings is designed precisely to help users get this dialed in.