[Support]: Ffmpeg continuously crashing #2858
Comments
|
It looks like instability in the stream is causing the RTMP relay to fail. It is very sensitive to bad data. I would try disabling RTMP with: rtmp:
enabled: FalseThe detect role cannot be removed and Frigate is adding it back in automatically. This is the role that decodes the video stream, does motion detection, and provides images for the frontend. Disabling detect just prevents the object detection from being run on motion. If you are still seeing failures that cause the recording segments to go missing, you can try running a separate process for record and detect like this: cameras:
yi_outdoor:
ffmpeg:
inputs:
- path: rtsp://192.168.1.XXX/ch0_0.h264
hwaccel_args: -c:v h264_v4l2m2m
roles:
- detect
- path: rtsp://192.168.1.XXX/ch0_0.h264
roles:
- record
detect:
enabled: False
# width: 1920
# height: 1080Lastly, your issues with the Coral sound like the normal power related issues. The Coral uses a lot of power for a USB device, so it is recommended to use a USB hub with an external power supply with a Pi. |
|
Hi @blakeblackshear. Thanks for the prompt reply. I have made the suggested changes but I still get crashes (although log messages are partly different): [2022-02-22 13:59:46] frigate.video ERROR : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 13:59:46] frigate.video ERROR : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 13:59:47] ffmpeg.yi_dome.record ERROR : [NULL @ 0x5585284030] missing picture in access unit with size 52
[2022-02-22 13:59:47] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 13:59:47] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 13:59:48] watchdog.yi_outdoor ERROR : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 13:59:48] watchdog.yi_outdoor ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 13:59:48] ffmpeg.yi_outdoor.detect ERROR : [rtsp @ 0x55bff34130] CSeq 4 expected, 0 received.
[2022-02-22 13:59:48] ffmpeg.yi_outdoor.detect ERROR : [h264_v4l2m2m @ 0x55c001b820] === poll unexpected TIMEOUT: events=0x43, cap buffers=20
[2022-02-22 13:59:57] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 13:59:57] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 13:59:57] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:07] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:07] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:00:07] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:17] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:17] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:00:17] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:18] watchdog.yi_outdoor INFO : No frames received from yi_outdoor in 20 seconds. Exiting ffmpeg...
[2022-02-22 14:00:18] watchdog.yi_outdoor INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:26] frigate.video ERROR : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 14:00:26] frigate.video ERROR : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 14:00:26] frigate.video ERROR : yi_outdoor: Unable to read frames from ffmpeg process.
[2022-02-22 14:00:26] frigate.video ERROR : yi_outdoor: ffmpeg process is not running. exiting capture thread...
[2022-02-22 14:00:27] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:27] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:00:27] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:36] watchdog.yi_outdoor ERROR : Ffmpeg process crashed unexpectedly for yi_outdoor.
[2022-02-22 14:00:36] watchdog.yi_outdoor ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 14:00:36] ffmpeg.yi_outdoor.detect ERROR : [rtsp @ 0x557c85f130] CSeq 4 expected, 0 received.
[2022-02-22 14:00:37] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:37] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:00:37] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:47] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:47] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:00:47] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:00:57] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:00:57] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:00:57] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:07] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:07] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:01:07] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:17] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:17] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:01:17] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:27] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:27] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:01:27] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:31] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:47024]
[2022-02-22 14:01:37] ffmpeg.yi_dome.record ERROR : rtsp://192.168.1.164/ch0_0.h264: Invalid data found when processing input
[2022-02-22 14:01:37] watchdog.yi_dome INFO : Terminating the existing ffmpeg process...
[2022-02-22 14:01:37] watchdog.yi_dome INFO : Waiting for ffmpeg to exit gracefully...
[2022-02-22 14:01:41] frigate.video ERROR : yi_dome: Unable to read frames from ffmpeg process.
[2022-02-22 14:01:41] frigate.video ERROR : yi_dome: ffmpeg process is not running. exiting capture thread...
[2022-02-22 14:01:47] watchdog.yi_dome ERROR : Ffmpeg process crashed unexpectedly for yi_dome.
[2022-02-22 14:01:47] watchdog.yi_dome ERROR : The following ffmpeg logs include the last 100 lines prior to exit.
[2022-02-22 14:01:47] ffmpeg.yi_dome.detect ERROR : [rtsp @ 0x557c812130] CSeq 5 expected, 0 received.
[2022-02-22 14:01:47] ffmpeg.yi_dome.detect ERROR : [h264_v4l2m2m @ 0x557c816840] === poll unexpected TIMEOUT: events=0x43, cap buffers=20Clips are still missing on both cameras (especially on the outdoor one, with a long crash every 2-3 minutes). Any other suggestions? I also observe (actually hear the fans) that the ffmpeg process gets stuck frequently with 100% usage of the cpu. I do not think that happened before (but I am not sure). Finally, regarding the coral, I forgot to say that the USB hub is powered, so that should not be the problem (I guess...). Thanks again for your help! |
|
Hi again. I have been playing around and it results that it was RSTP server's fault. I was using yi-hack-mstar v0.3.8 on one of the cameras and after updating to v0.4.4 and switching to an alternate server provided in this version everything seems to run much better. 15 minutes of uptime an no errors on the logs. Recordings also seem to be working fine. I will try to upgrade the version of the firmware of the other camera tomorrow. It is installed outside and now is dark here. I have however a couple of questions.
Thanks for everything. I will report back with the second camera issue, just in case somebody else is facing similar problems. |
|
Well... it crashed after 30 minutes. More work needed server-side... :/ |
|
@tonitonae Yeah, 30 minutes of no crashing is good. My Wyze cam also has crashes here and there just due to "meh" firmware unfortunately. As far as your questions go:
|
|
Hi @NickM-27. Thanks for the prompt reply. I am trying to figure out if something can be done server side to avoid these problems. Regarding 1, understood. As per 2: I checked the recorded clips and did not see anything on them, so it has to be something on the client (maybe specific to firefox, maybe not, I have to check). Thanks! |
|
Hi @blakeblackshear. I am experimenting a little bit more with the RTSP server. I do not know if I should say this, but now if has been running without issues for more than 80 minutes (compared to a maximum of half an hour yesterday it seems that something I have touched server-side has significantly improved its reliability). Anyway, what I have done now is monitoring frigate. I see no errors on the logs (good) but I see that the ffmpeg process in charge of the detection, even with the HW acceleration flags, is consuming 100% of one of the CPUs of the RPi 4. Is this the expected behavior? I still need to grab a USB A - USB A cable to see if I can use my HDD and the coral simultaneously, both connected to the USB hub (powered) but I wanted first to check if I can make the cameras and frigate work together. Thanks for everything! |
|
It simply takes a lot of resources to decode a 1080p video stream at 20fps. This is why it is recommended to use cameras with configurable frame rates and resolutions. The Pi can handle many more cameras if the streams are 720p at 5fps. In the config you posted, the detect resolution is commented out. This will cause ffmpeg to resize the stream to 720p before frigate processes it. This will reduce the CPU used by frigate, but increase the CPU used by ffmpeg. If you specify the resolution as 1080p in the detect config, it should lower the CPU used by ffmpeg and increase it for other processes. |
|
Thanks @blakeblackshear for the suggestion. I uncommented the resolution lines and now ffmpeg uses a lot less resources (around 15% of CPU, more or less). Another thing that I seem not to have understood completely is the detect role. You told me in a previous comment that it is mandatory and it is re-enabled automatically by frigate. In my config I had it with enabled: False and I had the warning in the log file about CPU detectors, so I thought that frigate was ignoring my False and using a True for that option. However, if I explicitly put True in the config file, then a frigate.detector (or similar, I did note it, sorry) process spawns taking 180-200% of CPU. Reverting it back to False that process does not appear. Obviously, it seems to be a difference between using True and False even if frigate is reenabling it. Does this mean that the detector process is always running but it does different things whether the config option is True or False? Thanks! |
|
The role is required, and frigate will add it to the list of roles under your input. Setting enabled to False tells frigate not to run object detection. It still needs the role assigned to an input so it knows which stream to use for motion detection and to display images and the live view in the UI. |
|
Ok, that makes sense. One last question (sorry for making too many of them :/): which is the purpose of motion detection? I do not see anything in the UI related to motion detection (I see the object detection and I even picked my wife while I was testing with the config XDDD) but I do not see how motion detection can be used. Is it described in the docs? I did not seem to find it, just about motion masks, but not motion detection itself. Thanks one more time. |
|
@tonitonae Motion detection has a handful of different uses. Frigate uses motion detection to decide when to run object detection. It isn't run on every frame just the frames where motion is detected. In 0.10.0 there is also the ability to set retain modes on 24/7 recording and events so that it only saves the recording segments that have motion. Currently this is only documented in the config file: https://docs.frigate.video/configuration/index |
|
Thanks @NickM-27. If it is only in the config that is why I was unable to find it. You did not mention and I did not see it so I imagine it is not possible, but I will just ask. In principle, I like to have 24/7 recordings, but it would be very nice if frigate could label the moments in which motion was detected similarly to what is done with object detection. Would that be possible? Maybe also creating a snapshot with a timestamp to easily identify where to look would be interesting. I am aware that there are many things to do, but I think that some kind of tagging for motion events in 24/7 recordings would be interesting. Thanks again for both the cool project and the help given here. Cheers! |
|
It is currently not supported, however it is something that has been requested somewhat often. I am not 100% sure, but believe all the pieces besides the UI are there. Not sure where it fits in to the roadmap, that would be a question for blakeblackshear. |
|
Ok, thanks for the feedback. Hope it is not very low in the todo list ;-) |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Hello @tonitonae, I'm running into same problems with Yi Hacked cameras (AllWinner and AllWinner2 hacks). Did you managed to solve issues and avoid ffmpeg crash? What are your best settings/configurations? Thanks |
Hi @outofsight. I do not have any AllWinner camera. I am using yi-hack v5 for my outdoor camera and yi-hack-mstar for my Dome camera. The first works more or less flawlessly (some frame drops here and there, but no crashes for weeks) but the MStar one crashes frequently. I think I have narrowed it down and crashes seem to be related to the switch of the camera to night vision when light from the outside gets too low. I have downloaded the source code of the RTSP server, but I did not have the time to work on it (too busy at the moment). If I find something interesting, I will let you know. Cheers! |
|
Hi @tonitonae Would you be able to post the config.yml file that you finally arrived at. Im having same issue thanks |
|
Anyone have good working ffmped args for the nyihack5 that works reliably. |
|
I got rid of the errors with the following line input_args: -rtsp_transport tcp |
Saved my night! 🦄 |
|
Saved my night too! I had the problem that frigate didn't record clips and snapshots anymore while live view was still working. Adding this option in /config/frigate.yml input_args: -rtsp_transport tcp solved this. THANK YOU!! |
Describe the problem you are having
Hi.
I am struggling to have frigate set up for 24/7 recording from two cameras on a RPi 4. I have been stucked as everybody else with the hwaccel bug for some time. With the latest release, now that is working (no more green screens, I can see the images from both cameras). Thanks!
My problem now is that I have a lot of errors on my logs and many frames dropped. Clips are not stored regularly every 1 minute. Instead, I have some clips of 10 seconds, other longer, sometimes no clip for 2-3 minutes. If I go to the recordings page in the UI, the largest aggregated length of a recording is of 8-9 minutes for an hour, which means that a lot of information is missing.
This is due, as the logs say, to the fact that ffmpeg crashes a lot (see below the errors). I do not know if this is due to my cameras or network setup. Which is weird is that this setup (network and cameras) has been working flawlessly most of the time with Synology Surveillance Station. I can also watch the streams with VLC and they seem quite fluent. I did not watch them for a long time, but I do not see those drops in a 3-5 minutes interval in which ffmpeg in the frigate container crashes several times.
As per the logs there is one thing that I do not understand. Whenever an error happens, the logs refer to a detect process which I, supposedly, have disabled in my config of frigate. I do not know if the name of the process does not actually mean that detection is being conducted or if the detection process is used for something else. I also see a warning at the beginning of the log about cpu detection not being recommended though I think that I am explicitly disabling it.
Sorry for the long message. I just wanted to provide enough context.
Thanks in advance for your help.
PS: both cameras are running unofficial yi-hack firmware, just in case someone has experience with those FWs.
PS2: maybe I should tweak some config of fmmpeg to make everything work properly, but I do not have any experience with that, so I do not know what to touch.
Version
0.11.0-3de1948
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
{"detection_fps":0.0,"detectors":{"cpu":{"detection_start":0.0,"inference_speed":10.0,"pid":216}},"service":{"storage":{"/dev/shm":{"free":64.0,"mount_type":"tmpfs","total":67.1,"used":3.2},"/media/frigate/clips":{"free":877255.6,"mount_type":"ext4","total":984280.7,"used":67006.4},"/media/frigate/recordings":{"free":877255.6,"mount_type":"ext4","total":984280.7,"used":67006.4},"/tmp/cache":{"free":999.7,"mount_type":"tmpfs","total":1000.0,"used":0.3}},"temperatures":{},"uptime":3263,"version":"0.11.0-3de1948"},"yi_dome":{"camera_fps":2.76,"capture_pid":226,"detection_fps":0.0,"pid":223,"process_fps":1.6,"skipped_fps":0.0},"yi_outdoor":{"camera_fps":5.0,"capture_pid":225,"detection_fps":0.0,"pid":220,"process_fps":5.0,"skipped_fps":0.0}}Operating system
Other Linux
Install method
Docker Compose
Coral version
CPU (no coral)
Network connection
Wireless
Camera make and model
Yi Outdoor and Yo Dome 1080p
Any other information that may be helpful
I do have a coral device (USB) but I can not connect it to my RPi as it hangs for some reason :/
I use this box:
https://www.amazon.es/gp/product/B095WD7BTW
Which has a sort of USB bridge to connect the 2.5 HDD to one of the USB 3 ports. If I connect the Coral to the other USB 3 port, then it complains about power during boot. I have a USB hub:
https://www.amazon.es/gp/product/B00TPMEOYM
But, if I connect it to the second USB port, then the RPi does not boot. Or, If I connect it when the Rpi is running, then if hangs at some point. It seems like some kind of interference with the "USB bridge". I need more testing (maybe connecting the HDD to the hub directly with a USB A - USB A cable) but using coral is not an option right now.
Regarding the OS, I use DietPi 64bit.
Thanks for your help.
The text was updated successfully, but these errors were encountered: