-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Camera stops recording after a few days #209
Comments
Are you using TCP or UDP to talk with the cameras? (The default is TCP, so it's TCP unless you picked UDP within the camera setup dialog.)
This says the camera decided to end the stream for whatever reason. I don't think there's anything Moonfire could do differently here.
I might add some more debug logging to narrow this down. Not sure how familiar you are with RTSP, but it's layered on top of another protocol, RTP. RTP packets all have a When this happens, Moonfire suspects it's due to a bug in old versions of a popular RTSP server implementation called live555. In short, the server doesn't tear down TCP sessions properly, and the camera can end up sending several of them over the same connection, which cascades into problems with overload and then another bug that causes the connection to become corrupt after the Moonfire tries to avoid this cascade by waiting for the old session to go away before trying again, which is why you see this message:
What happens after that? There should be another attempt around 20:10:44. |
I'm using TCP explicitly in the config
Nothing, that's the end of the logs
If the camera kills the stream, that's ok for me, as long as Moonfire retries indefinitely to reconnect, which doesn't seem to be the case here. Is that expected? |
😞 It kept running but didn't log any more? Was this the only camera Moonfire was configured to record?
No, that part is definitely broken then. This sounds similar to #184, which I'd thought I'd fixed. I'll take another look. |
Reread through the code in question, nothing's jumping out at me yet. If you don't mind, it might help to enable more logging. Where there are |
Yeah, it kept running and recording the second camera. Just no recordings/logs for that one anymore. I just enabled the logs, and I'm now getting logs like this:
Is that what you're looking for? If so, I'll send you everything once it happens again |
Ah, second problem: the second camera has actually stopped recording since I restarted Moonfire on March 19th: As you can see, there's a 0/1 second recording, then nothing. I've restarted multiple time just right now, but the recording won't restart. The camera is working/streaming fine outside of Moonfire. I attached the logs since I restarted a few minutes ago. Probably a different bug, though? |
Ahh! Got it. Those logs helped, and I believe both cameras' problems are due to the same bug. From this log line:
I see that it reached this code here: in which it sees a strange And we see that Moonfire is waiting for this session to go away:
but it never does [edit: removed confused ideas based on looking at the wrong version of the code] because you're running Moonfire v0.7.1, which predates my fix to #184. Also, your camera isn't advertising itself as using a buggy version of live555 (even though it is using such a version, or some other software with a similar bug), or we'd have seen this line logged: Anyway, I think there are several changes to make to improve this situation:
In the meantime, a couple ways you can work around this problem:
|
Ah, glad it helped! Thanks for taking for the time to chase that bug! Unfortunately, I have little to no knowledge to help you to fix it... By this time, I'm working on (I already have a working PoC) a small gateway to monitor Moonfire through Zabbix, with automatic discovery, thanks to Moonfire API. That allows me to trigger a problem when a camera has not any recording for 5 minutes. |
Hi Robin, |
Hi John,
The second part is a Zabbix template to connect Zabbix to it. That's the end result: I plan to release it shortly |
No worries. If you'd like to learn more (about Rust, RTSP, etc.), let me know. Otherwise, I've got this.
Neat! I don't know Zabbix, but in general: I'm open to adding support for white-box monitoring to Moonfire itself also. We can discuss more in a separate issue. I've been meaning for a while to export metrics in the Prometheus exposition format. It looks like Zabbix supports this format also. |
Huh. On second (third? fourth? I've lost track) look, I have code that's supposed to do this here: and it seems to be working in my best attempt to reproduce locally (an old Reolink camera with this bug, a hardcoded change to |
Simply push a docker image on a different name, I'll run and try it until it fails. As you can see, it can take 1 to 2 weeks.
Nice to hear that! I'll have a deeper look then. |
Oh! Moonfire NVR 0.7.1 is older than I realized. It predates my fix for #184. I should have been looking at Retina version v0.3.4 rather than latest when trying to figure this out. That explains why it didn't recover after 65 seconds. If you upgrade to the latest Moonfire NVR (note you'll need to adapt your The other items in my checklist still apply. |
I'll do the upgrade (tomorrow?) and see if it helps. BTW, would you mind making use of Github releases? That makes it much easier to track new releases. |
I've updated both of my installations to 0.7.3. I'll update when/if anything bad happen w/ it. |
As described here: scottlamb/moonfire-nvr#209 (comment)
Newest commit addresses each of the checklist item. I plan to make a new release after fixing #206 and possibly #217. Regarding this item:
I adjusted the |
v0.7.4 released. |
Describe the bug
One of my camera (the same as in #197) stops recording after a few days with the following logs:
Expected behavior
The camera should record indefinitely, or restart if the stream dies for some reason.
Screenshots
![Screenshot from 2022-03-30 10-31-38](https://user-images.githubusercontent.com/9640353/160788097-9270d2d1-3469-44e7-973d-55c5da8aabd1.png)
![Screenshot from 2022-03-30 10-31-25](https://user-images.githubusercontent.com/9640353/160788109-da8f52ba-cf3f-4cfe-ad3b-27539904fa4c.png)
![Screenshot from 2022-03-30 10-33-59](https://user-images.githubusercontent.com/9640353/160788239-9512a951-8c12-41bb-9ca0-7ecfe1dbb70f.png)
Server (please complete the following information):
Camera (please complete the following information):
Model: Trendnet TV-IP311PI
Firmware Version: V5.0.1 140303
The text was updated successfully, but these errors were encountered: