Restart of recording streams #1713
Comments
|
What cameras do you have? There is already something in place to check if the recording stream needs to be restarted. |
|
I have those devious Reolink cameras that are known to have issues with RTMP streams. I found a (mostly) working settings for ffmpeg input args from another issue some time ago, but they still seem to fail at some point. But a pure frigate restart will recover. Now that you said that there is a check for the streams it's possible that sometimes the recovery works and sometimes not. I'm not 100% sure but I think both my Reolinks stopped working last time in the same minute (and didn't recover) while my other brand cameras kept working. Unfortunately docker logs for that aren't available now as I just upgraded the image. I think I'll be removing the nightly restart of frigate and try to recreate the issue again. |
|
Do you have the rw_timeout option in your config? |
|
No, actually, I had that commented out as I wasn't sure what that does and otherwise it worked perfectly. - -rw_timeout
- "5000000" |
|
Those parameters tell ffmpeg to timeout if it is no longer receiving data from the rtmp feed. It seems to resolve this issue for Reolink cameras. If they aren't working, provide your config and logs. |
|
Hey, I got one of the cameras failing again (after frigate uptime of 2 days), below relevant config for the camera, and below that, the relevant log entries. Edit: I'm going to try changing Reolink streams to http instead of rtmp according to what is said in #1467 to see if it's more stable. "cameras": {
"back": {
"best_image_timeout": 60,
"detect": {
"enabled": true,
"fps": 7,
"height": 480,
"max_disappeared": 35,
"width": 640
},
"ffmpeg": {
"global_args": [
"-hide_banner",
"-loglevel",
"warning"
],
"hwaccel_args": [
"-hwaccel",
"vaapi",
"-hwaccel_device",
"/dev/dri/renderD128",
"-hwaccel_output_format",
"yuv420p"
],
"input_args": [
"-avoid_negative_ts",
"make_zero",
"-fflags",
"nobuffer+genpts+discardcorrupt",
"-flags",
"low_delay",
"-strict",
"experimental",
"-analyzeduration",
"1000M",
"-probesize",
"1000M",
"-rw_timeout",
"5000000"
],
"inputs": [
{
"global_args": [],
"hwaccel_args": [],
"input_args": [],
"path": "rtmp://redacted",
"roles": [
"record",
"rtmp"
]
},
{
"global_args": [],
"hwaccel_args": [],
"input_args": [],
"path": "rtmp://redacted",
"roles": [
"detect"
]
}
],
"output_args": {
"detect": [
"-f",
"rawvideo",
"-pix_fmt",
"yuv420p"
],
"record": "-f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy",
"rtmp": [
"-c",
"copy",
"-f",
"flv"
]
}
},
"ffmpeg_cmds": [
{
"cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000 -i rtmp://redacted -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/back-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://127.0.0.1/live/back",
"roles": [
"record",
"rtmp"
]
},
{
"cmd": "ffmpeg -hide_banner -loglevel warning -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer+genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000 -i rtmp://redacted -r 7 -s 640x480 -f rawvideo -pix_fmt yuv420p pipe:",
"roles": [
"detect"
]
}
],
"live": {
"height": 720,
"quality": 8
},
"motion": {
"contour_area": 74,
"delta_alpha": 0.2,
"frame_alpha": 0.2,
"frame_height": 180,
"mask": [
"0,71,162,62,231,31,640,114,640,0,0,0"
],
"threshold": 25
},
"mqtt": {
"bounding_box": true,
"crop": true,
"enabled": true,
"height": 270,
"quality": 70,
"required_zones": [],
"timestamp": true
},
"name": "back",
"objects": {
"filters": {
"bird": {
"mask": null,
"max_area": 4000,
"min_area": 0,
"min_score": 0.5,
"threshold": 0.79
},
"cat": {
"mask": null,
"max_area": 24000000,
"min_area": 0,
"min_score": 0.5,
"threshold": 0.79
},
"dog": {
"mask": null,
"max_area": 24000000,
"min_area": 0,
"min_score": 0.5,
"threshold": 0.79
},
"person": {
"mask": null,
"max_area": 24000000,
"min_area": 0,
"min_score": 0.5,
"threshold": 0.7
}
},
"mask": "",
"track": [
"person",
"dog",
"cat",
"bird"
]
},
"record": {
"enabled": true,
"events": {
"enabled": true,
"max_seconds": 300,
"objects": null,
"post_capture": 5,
"pre_capture": 5,
"required_zones": [],
"retain": {
"default": 5,
"objects": {}
}
},
"retain_days": 5
},
"rtmp": {
"enabled": true
},
"snapshots": {
"bounding_box": true,
"clean_copy": true,
"crop": false,
"enabled": true,
"height": 500,
"quality": 70,
"required_zones": [],
"retain": {
"default": 5,
"objects": {
"person": 5
}
},
"timestamp": false
},
"timestamp_style": {
"color": {
"blue": 255,
"green": 255,
"red": 255
},
"effect": null,
"format": "%m/%d/%Y %H:%M:%S",
"position": "tl",
"thickness": 2
},
"zones": {}
},[2021-09-08 06:34:03] watchdog.back INFO : No frames received from back in 20 seconds. Exiting ffmpeg... |
|
I've now had http instead of rtmp for Reolink cameras. It's been working really solid for 6 days. If this is solution, maybe the Reolink specific instructions should be changed. I'll keep testing for a few days. |
|
@salleq please follow up on the http streams and whether they have been stable for you. I've also been looking for a solution for the Reolink streams for a while also. I have the rw_timeout in place on ffmpeg but it doesn't seem to notice that the stream is dead. I was following up on this in #1406 but that issue is closed. This morning this has happened again. Commands and logs below: You can see below that the ffmpeg processes are still running (with the rw_timeout) option, but files are not being created in /tmp/cache anymore. The debug screen in the gui obviously still shows "camera fps" for this camera, because the detect stream is still running. You can see in the logs where the camera seems to be having all kinds of issues around 2021-09-16 18:39:1 which is the last time the file for it in /tmp/cache got written to. It seems like frigate is able to determine that the detect stream has died, but isn't as "aware" that the record stream has died. I assume this is because the detect stream is "processed" further down the pipeline, whereas the record streams are just written to disk. I too have been toying around with creating a script that verifies for each camera whether there are "current" files in /tmp/cache, and if not, bounces the entire frigate container instead of just bouncing it once per day or something like that. My idea was that I should be able to detect if a record stream has died fairly quickly by looking if there are any files for the current minute in /tmp/cache. If there are no current files for one of the cams, I would have to just restart the entire container. Not ideal, but better than missing a recording because the record stream died a few hours/days ago. @blakeblackshear are we missing anything here where frigate (or ffmpeg) should be able to find out that a record stream has died? root@943583cd85dc:/tmp/cache# ps -ef | egrep -i drive root@943583cd85dc:/tmp/cache# ls -l /tmp/cache [2021-09-16 18:39:18] watchdog.garage_cam INFO : No frames received from garage_cam in 20 seconds. Exiting ffmpeg... |
|
Hey, have to admit that it didn't actually "work solidly" for 6 days. I had a lot of restarts due to my restart scripts, but I wasn't looking at them...amateur... I looked at differences in their settings (not Frigate but the cameras themselves), and noticed that the front yard, which was crashing more often, had higher bitrate set in "Fluent" stream (which I use for detection). I know, it shouldn't matter as the recording stream was the one that kept crashing. However, now after a couple of days it seems to have been working better. It might not be the end-of-all-crashes fix, as I've seen back yard camera crash as well, but a lot less frequently. Next I'm going to try switching back to the "Clear" stream for detection, too. |
|
Hey, I've had frigate running for two weeks now without forced restarts. The only times I get any errors for my Reolink cameras have been when they have their weekly reboots done, and frigate can recover from that. So I could say that playing around with the bitrates and having all the Reolink specific ffmpeg flags will make the streams a lot more stable. |
|
Care to share your working config? |
|
@salleq I would love to update the Reolink specific page in the docs with some specifics. Reolink problems may be the most common issue I see. |
|
Here's my config for my Reolink cameras at the moment (a RLC-410-5MP and a RLC-520) cameras:
reolink:
ffmpeg:
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420p
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer+genpts+discardcorrupt
- -flags
- low_delay
- -strict
- experimental
- -analyzeduration
- 1000M
- -probesize
- 1000M
- -rw_timeout
- "5000000"
inputs:
- path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password
roles:
- record
- rtmp
- path: http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=username&password=password
roles:
- detect
output_args:
record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
detect:
width: 640
height: 480
fps: 7In addition my Reolinks' configs are as follows:
Note: I've had these also working with only having the "clear" stream, but I can't guarantee that it works as well. In that case I would only use one input for all roles (channel0_main). Might try that out again though as detection isn't as good as I'm hoping for it to be with these settings. One setting that clearly had an effect with the streams working better was lowering the bandwidth on "fluent" stream. I had it at 512kbps earlier and it kept crashing every two days or so. |
|
@salleq it is also possible to record the audio? Thanks |
|
… On Wed, Oct 6, 2021 at 5:50 PM Keltere ***@***.***> wrote:
@salleq <https://github.com/salleq> it is also possible to record the
audio? Thanks
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1713 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AC3I7673HCK4UMUKB572SODUFTABXANCNFSM5DO7SJTA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
When I follow your config template for just one of my cameras I get: "Error parsing config: extra keys not allowed @ data['cameras']['reolink']['detect']['width']" This happens at start up. Everything is the same as your config file for the camera plus the camera stream settings. Any Idea what I am doing wrong? |
You are not running the latest release. |
Oh. I installed through community applications on Unraid. It said I was up to date. I'll look I to somehow getting it updated manually or something else. Thanks. That's probably the cause of all my problems I have been having. |
|
Well Now I am running the latest release 0.9.1-800F33E and I am unable to get my Reolink Camera working with the above config. I am getting the following errors in the log: My Config file is as follows: I had cameras working with a different config on the previous release but they were causing me all sorts of problems after several hours (1-2 second recordings, broken live streams). Now I cannot get the cameras working at all with the recommended config Any ideas? I have the Reolink RLC-410-5MP just like mentioned above. |
|
Thanks for posting the http feed example. I haven't tested for long time but the freezing/shuttering recording issue with Reolink 410 feed seems to have resolved. I can even record with the default camera settings (30fps and 6144kbps) via Frigate without the issue I was having. |
|
How do I use the flag to copy audio also? I tried following but the debug still shows "-c copy -an" (....1 -strftime 1 -c copy -an /tmp/ca....) which I think only copies the video. output_args: |
|
found the answer - |
|
Any idea why I might be getting a 404 not found with the RLC 520 and the links (edited to match my IP, username and password, of course) provided above? |
|
Which firmware version do you have? I tested the links again and in browser I get a .flv download from the link. |
|
That was exactly my issue! Thank you! Just needed to get the newest firmware. |
|
If you've got
i.e. for this path ...use this detect: Otherwise, for the fluent (640x480) stream use this path: But IMO the balanced stream is a better choice for detect, as for only a small resolution bump the live and birdseye views are noticeably clearer. @blakeblackshear can you please correct the docs? |
|
Does this setup? particularly the http access require Firmware version 3.0? http access does not work, I only have main and 1 sub rtsp stream And according to the faq The portion of the config PS: I have setup the sub stream to be 4 fps on the camera, I found it closer to 5 than the default 7, and matched the frigate to it. My detect path is setup as Any guidance is greatly appreciated. |
I don't have a Reolink NVR but looks like that does not work the same way as using the cameras standalone. To me it seems you're accessing only one RTSP stream of the NVR instead of having stream per camera.
I don't know if it's possible without removing a camera from NVR (is it available in network even though connected to NVR?), but I would definitely try one camera first standalone in frigate and if you can get that work, add more. By the way your NVR password is in cleartext in the screenshot, you might want to clean that. |
|
Thanks @salleq for the response and also point out my clear-text password. As for the camera, I realize my post was not very clear. I'll try to detach one of the cameras and test in standalone mode, but having all of them in standalone mode is not feasible due to POE / power adapter needs. Thanks |
|
Note: Looks like the only diff are that |
|
It is worth noting that when I restart the frigate container, at first for a few minutes, some of the camera screens are garbled. This does not happen to all the cameras, but the ones that show up garbled, persist being garbled for several minutes Also the following option on input_args |
|
According to this post, Reolink cameras use a Live555 RTSP server from 2013, which doesn't handle restarting the stream properly. I've confirmed the Reolink RLC-410W camera I own advertises using "LIVE555 Streaming Media v2013.04.08".
Basically, if your frigate client drops the connection, the camera doesn't realize it, and keep sending the "old" stream. When frigate reconnects, the camera starts sending a "new" stream, together with the old one, which confuses the client. Sending two streams over one TCP connection is a violation of the RTSP, so it's a Reolink camera issue, not a client issue. If we want to be able to use RTSP with Reolink cameras, it seems Reolink has to issue a firmware update with a recent Live555 server in it. I've contacted their support about it. |
|
FYI, I've been working with Reolink support on this for months. They finally came back about a week ago and I got the below response back. Even though they're still selling the RLC-410-5MP on their website today they aren't planning to fix it (at least not publicly). I had one of my cams get damaged by a pressure washer recently and purchased a Loryta at Blake's suggestion. I can firmly say, its worth the money. The streams are SO MUCH more stable and configurable. If you value your time and can afford it, I would suggest throwing the Reolinks in the trash and purchase some Loryta's from Amazon. I will be slowly replacing each of mine as they die or as I get tired fooling with them trying to make them stable. I've easily wasted more money worth of my time fooling with the Reolinks than I've saved by purchasing them in the first place. I just asked our product manager again about this upgrade matter. He says that he has contacted the original factory that produces the camera's chip. But even the firmware is made, we can't make it openly available on the website. Anyway, thanks for your important feedback and suggestion. If the firmware is feasible, we will contact you again. |
|
PS...just imagine how many support tickets and time has been wasted within the frigate project alone trying to figure out how to make these cams stable. :( :( Give up. |
|
I wish I'd discovered this thread in August before my window to return my four reolink cameras to Amazon had closed! Pretty bummed to learn Reolink just left a massive bug like this sitting around, unpatched and unacknowledged. I was half considering writing some kind of watchdog to monitor the streams and reboot the cameras before just following the advice here and switching over to the http feeds for my three RLC-520s and single RLC-820A. I guess the "bonus" (if you want to call it that) is that at least now I can tap the |
How are other software able to deal with ReoLink cameras? I've used Shinobi and hardware based NVR's and none of them have the problem that frigate has with Reolink cameras. Is there something that can be done on the software side to work around this issue? |
|
I would like to kindly point out my findings on Reolink cameras if this is not already known: Reolink RLC-410-5MP IP camera firmware unpacker and reverse engineered technical details. Maybe someone is able to patch the camera Linux system so that the Live555 server is updated (... or even replaced with a completely different streaming-server). |
|
So, I spent quite some time getting this all set up for a couple Reolink cameras, an E1 Outdoor, and an E1 Pro. The E1 outdoor works well with the http config above, but the E1 Pro does not have a http stream, and is rtsp only. I stumbled upon this github issue - #2334 So, i tried using the |
This solution also worked for all of the cameras in my setup, including the Amcrest ASH47-W and AD410. |
|
Hello, I'm just leaving this here in case it could help someone : after hours of debugging I found out http streams don't work if you have special characters in your camera password (I had "$" and couldn't connect to http stream). |
This is the case for RTSP streams as well |
OMFG, thank you! Fixed it for me. |
|
unfortunately, I recently bought 4 reolink 820A's on Amazon Prime day without doing my homework! Trial and error with Frigate now has Frigate log not showing errors but my camera feed is blank green - any thoughts? - detect- clips |
Have you looked at https://docs.frigate.video/configuration/camera_specific#reolink-410520-possibly-others ? Works well for my 511WA You might also want to make sure you're on the latest firmware (compare with reolink download center) as I had to download a firmware update from there to get it to work optimally as there's new iframe and other options Also if you have green screen then you'll have ffmpeg errors |
|
thanks for your help - I now have it working on the sub stream but cannot
access the cameras directly at the moment to fiddle with the resolutions as
I am away.
…On Mon, 22 Aug 2022 at 10:20, Nicolas Mowen ***@***.***> wrote:
unfortunately, I recently bought 4 reolink 820A's on Amazon Prime day
without doing my homework! Trial and error with Frigate now has Frigate log
not showing errors but my camera feed is blank green - any thoughts?
Have you looked at
https://docs.frigate.video/configuration/camera_specific#reolink-410520-possibly-others
? Works well for my 511WA
You might also want to make sure you're on the latest firmware (compare
with reolink download center) as I had to download a firmware update from
there to get it to work optimally as there's new iframe and other options
—
Reply to this email directly, view it on GitHub
<#1713 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQEIO6V7EG3AYADCOPDKVEDV2LBUPANCNFSM5DO7SJTA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
*Michael Brooks*
|
|
Just thought I would post since it took a bit of work to collect information from several posts to get the older Reolink RLC-411 (IPC_3816M) and RLC-420-5MP (IPC_51516M5M) working with both audio & video. I like breathing new life into old hardware. |
and others
Describe what you are trying to accomplish and why in non technical terms
It seems that detect streams are somewhat recovered/restarted if there's an issue in the stream. This does not seem to affect recording streams though. I've come across few times that I still get detections correctly and events get created, but since recording is crashed, I cannot view any said events (except for the low resolution thumbnail)
Describe the solution you'd like
Recording streams should be restarted as well if they crash
Describe alternatives you've considered
Right now I have a daily restart of frigate as this issue seems to come up after a couple of days. I've also tried to trigger a frigate restart from logs if errors show up, but not all stream/connectivity errors should trigger a frigate restart (for instance if a camera is offline for any reason)
The text was updated successfully, but these errors were encountered: