Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Camera Stream Stops #364

Closed
dmkjr opened this issue May 4, 2022 · 31 comments
Closed

Camera Stream Stops #364

dmkjr opened this issue May 4, 2022 · 31 comments

Comments

@dmkjr
Copy link

dmkjr commented May 4, 2022

I'm currently running 1.3.8 but have been experiencing this problem for a few versions now. Randomly, every few days, the streams will just stop. The RTSP stream is functioning fine, I can use VLC to run it. However, Frigate stops ingesting the data.

I have opened up an issue here, as well. Here is a response from one of the developers from Frigate after reviewing the data I submitted.

"Frigate is losing a connection with the camera server. The wyzebridge is returning 404 not found implying it is having an error. What do the logs for that software look like?

All cameras stopping at the same time further points to the idea that it is some issue with the wyzebridge server and then it won't accept frigates reconnection attempt"

It appears Wyze Bridge is returning a 404 error on the streams.

0] ffmpeg.garage.detect ERROR : rtsp://10.0.10.7:8554/garage-cam: Server returned 404 Not Found [2022-05-03 14:51:10] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-05-03 14:51:10] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-05-03 14:51:20] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-05-03 14:51:20] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-05-03 14:51:20] ffmpeg.garage.detect ERROR : [rtsp @ 0x5634dfcebdc0] method DESCRIBE failed: 404 Not Found [2022-05-03 14:51:20] ffmpeg.garage.detect ERROR : rtsp://10.0.10.7:8554/garage-cam: Server returned 404 Not Found [2022-05-03 14:51:20] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-05-03 14:51:20] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-05-03 14:51:30] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-05-03 14:51:30] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-05-03 14:51:30] ffmpeg.garage.detect ERROR : [rtsp @ 0x558aa5a1bdc0] method DESCRIBE failed: 404 Not Found [2022-05-03 14:51:30] ffmpeg.garage.detect ERROR : rtsp://10.0.10.7:8554/garage-cam: Server returned 404 Not Found [2022-05-03 14:51:30] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-05-03 14:51:30] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-05-03 14:51:40] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage.``

Any idea of what I could try?

@dmkjr
Copy link
Author

dmkjr commented May 4, 2022

not sure why the link didn't post. blakeblackshear/frigate#3185

@mrlt8
Copy link
Owner

mrlt8 commented May 4, 2022

That is strange.

Just to confirm, is the rtsp stream still available in VLC when frigate stops reading the stream or does it stop working as well?

Anything in the wyze bridge logs?

@mrlt8
Copy link
Owner

mrlt8 commented May 4, 2022

Could it be -stimeout 5000000...?

The cams will occasionally stop returning frames but the bridge will keep retrying for about 20s (RTSP_READTIMEOUT=20s) before restarting the connection.

@dmkjr
Copy link
Author

dmkjr commented May 5, 2022

@mrlt8 sorry for the slow reply. Meetings all day. I don't have a timeout in the environment. I have one camera (the Wyze Car) that is always offline and keeps trying to connect for days. Last night, I put it in the block filter since I really will never put this into Frigate anyway.

Yes, VLC was still working, even after Frigate stopped grabbing the video streams.

Should I add a timeout as you mentioned above?

@mrlt8
Copy link
Owner

mrlt8 commented May 5, 2022

You have the the stimeout set in your frigate config under input_args.

If that is the cause then you may want to play around with that either in frigate or in the bridge with RTSP_READTIMEOUT.

@dmkjr
Copy link
Author

dmkjr commented May 5, 2022

Does that explain 404 error? I would think if it was timeout it would be a forbidden?

@mrlt8
Copy link
Owner

mrlt8 commented May 5, 2022

Any chance you can post the logs from the bridge? Would make it easier to know what's going on.

May also want to set RTSP_LOGLEVEL=debug so we can see additional info about the connection attempts.

@dmkjr
Copy link
Author

dmkjr commented May 5, 2022

Sure thing, I can do that. Is that set as an environment variable within the bridge?

@dmkjr
Copy link
Author

dmkjr commented May 5, 2022

@mrlt8 I have debugging on. I'll let it do it's thing. Here is something I've seen already, what exactly does this mean?

2022/05/05 00:51:22 [Dogs Room] WARNING: FPS param mismatch (avRecv FPS=10)

@mrlt8
Copy link
Owner

mrlt8 commented May 5, 2022

It's another weird wyze bug where the cam is reporting a certain frame rate but is actually returning another in your case, it's probably reporting 15 FPS but returning 10 FPS. This was causing some cameras to drift a few versions ago, but we should be able to compensate for it now.

btw, are the cloud recordings for that camera playing back slower/faster in the wyze app?

@dmkjr
Copy link
Author

dmkjr commented May 5, 2022

Appears to be playing back at normal speed are you referencing time to access or just video playback in general?

@mrlt8
Copy link
Owner

mrlt8 commented May 5, 2022

video playback in the events tab in the app.

@dmkjr
Copy link
Author

dmkjr commented May 6, 2022

@mrlt8 video playback in the events tab appears fine.

@dmkjr
Copy link
Author

dmkjr commented May 9, 2022

@mrlt8 as a quick update, I still haven't figured out what is causing the issue yet. Frigate is still up and running but as I just logged in, the streams are frozen at two days ago.

[2022-05-08 22:12:13] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-05-08 22:12:23] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-05-08 22:12:23] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-05-08 22:12:23] ffmpeg.garage.detect ERROR : [rtsp @ 0x56196ce33dc0] method DESCRIBE failed: 404 Not Found [2022-05-08 22:12:23] ffmpeg.garage.detect ERROR : rtsp://10.0.10.7:8554/garage-cam: Server returned 404 Not Found [2022-05-08 22:12:23] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-05-08 22:12:23] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-05-08 22:12:33] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-05-08 22:12:33] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-05-08 22:12:33] ffmpeg.garage.detect ERROR : [rtsp @ 0x5633068b3dc0] method DESCRIBE failed: 404 Not Found [2022-05-08 22:12:33] ffmpeg.garage.detect ERROR : rtsp://10.0.10.7:8554/garage-cam: Server returned 404 Not Found [2022-05-08 22:12:33] frigate.video ERROR : garage: Unable to read frames from ffmpeg process. [2022-05-08 22:12:33] frigate.video ERROR : garage: ffmpeg process is not running. exiting capture thread... [2022-05-08 22:12:43] watchdog.garage ERROR : Ffmpeg process crashed unexpectedly for garage. [2022-05-08 22:12:43] watchdog.garage ERROR : The following ffmpeg logs include the last 100 lines prior to exit. [2022-05-08 22:12:43] ffmpeg.garage.detect ERROR : [rtsp @ 0x55ced7220dc0] method DESCRIBE failed: 404 Not Found [2022-05-08 22:12:43] ffmpeg.garage.detect ERROR : rtsp://10.0.10.7:8554/garage-cam: Server returned 404 Not Found [2022-05-08 22:12:43] frigate.video ERROR : garage: Unable to read frames from ffmpeg process.

@mrlt8
Copy link
Owner

mrlt8 commented May 9, 2022

Do you have audio enabled?
Could you post the logs from the wyze bridge?

@dmkjr
Copy link
Author

dmkjr commented May 9, 2022

Do you have audio enabled?
Could you post the logs from the wyze bridge?

no audio

See attached. Let me know if that helps.
_wyze-bridge_logs.txt

@dmkjr
Copy link
Author

dmkjr commented May 9, 2022

@mrlt8 does that help or do you need another log?

@mrlt8
Copy link
Owner

mrlt8 commented May 9, 2022

The logs are showing the no one is publishing to path error which means the bridge is not publishing the stream from the camera for some reason...

Can you post the regular logs from the wyze bridge? It should show the cameras going down and/or if they go up again.

@dmkjr
Copy link
Author

dmkjr commented May 9, 2022

@mrlt8 sorry for my ignorance here, where are those logs located that I can pull?

@mrlt8
Copy link
Owner

mrlt8 commented May 9, 2022

They should just print out when you run docker to start the bridge or in the logs tab under the wyze bridge add-on if running in home assistant mode.

@dmkjr
Copy link
Author

dmkjr commented May 9, 2022

@mrlt8 See attached full log file.
wyze-bridge.log.zip

@mrlt8
Copy link
Owner

mrlt8 commented May 10, 2022

Thanks! It seems like of the six cams, 3 go down (man-cave, garage, and dogs-room) after about an hour and don't come back? But I don't see why.

Could you try turning down the log level on RTSP to info and turning on debug_frames:

      - RTSP_LOGLEVEL=info
      - DEBUG_FRAMES=true

@dmkjr
Copy link
Author

dmkjr commented May 10, 2022

I will try this now and report back. One thing I was thinking about, I would need to check logs to verify, you don’t think PiHole is somehow interfering with the connection do you? I looked for rejected DNS entries and didn’t see any blatant errors. I’m using Unbound as my upstream target.

@mrlt8
Copy link
Owner

mrlt8 commented May 10, 2022

I don't think so. I believe those are dns based blockers, but from what I could see in your logs, the bridge was able to connect directly to most of the cams over lan, except for the camper which connected over p2p.

@dmkjr
Copy link
Author

dmkjr commented May 10, 2022

The camper camera is at a remote location. The rest of the cameras are on my LAN.

@dmkjr
Copy link
Author

dmkjr commented May 10, 2022

@mrlt8 here are some more logs after running for awhile after adding debug_frames.
wb-may10.log

@mrlt8
Copy link
Owner

mrlt8 commented May 11, 2022

Thanks! I see camper go down for about 2 min, but other than that the other cams seemed to be ok? Did they go down in frigate during that time?

@dmkjr
Copy link
Author

dmkjr commented May 11, 2022

It's still up as of right now. The camper is at the lake and service is spotty. Sometimes that feed may drop. I rebooted the system last night to get it back up. It usually works for a day or so and then stops. When it stops, I'll pull logs again and provide them.

@dmkjr
Copy link
Author

dmkjr commented May 12, 2022

@mrlt8 It went down at 4:28am this morning. Here is the logs.
wb-may12.log

@mrlt8
Copy link
Owner

mrlt8 commented May 13, 2022

Which cams were you having trouble with in frigate? I see /back-deck took a little long to come back, but the other two seemed to come back within a minute.

@dmkjr
Copy link
Author

dmkjr commented May 21, 2022

@mrlt8 So.... I waited a few days to make sure this was it. Apparently...... shrinks into the fetal position while typing..... my VM ran out of storage space from a bad config within Frigate. Since changing 24/7 recordings to 1 day, it appears to have resolved the issue. Frigate was stopping the connections because it had no physical space left to write. I think Frigate should implement a check to display in the GUI. Going to recommend that now.

@dmkjr dmkjr closed this as completed May 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants