-
Notifications
You must be signed in to change notification settings - Fork 169
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High CPU Use #64
Comments
This usually happens on Celeron processors. I don't know why. On the Raspberry 3 the CPU load is 10% with 4 cameras |
Mine is an i3 proccessor, and load starts low with 4 cameras but when you see the image for a while begin to increase. Then, keep high until home assistant restart. |
Ich have the same problem with a Intel Xeon E-2144G processor (Home Assistant on a proxmox VM). |
I run Hass io on 2 Rpi's ( 1 raspberry pi 3b and 1 raspberry pi 4b) both give a CPU load of around +95% for 5 camera's. |
Hi! The same thing on i3 processor. After restart HA it use CPU around 14-17%. After some time and after cameras access - it increase up to 60%. I have 4 cameras in integration. If you need more technical info (logs, htop, etc) - just say :) |
I need answers from every one:
Configuration > Integrations > WebRTC > 3 dots > Reload will restart binary app |
Here my findings to your questions:
|
I've noticed that moving webRTC cards insice lovelace will cause the card to be reloaded adding CPU usage and not freeing previous used CPU (and memory). |
I am having this same issue on a RPI 4. 1 camera, multiple users. Unifi G4 Doorbell. It takes about 12-24 hours to eventually slow down my system, but the WebRTC stream, NodeRed and Home Assistant start fighting for CPU each using about 33% (~1.25 cores each) of the RPIs computing power. For now I am going to replace the WebRTC card with the default picture one and move the WebRTC one to a special view for it so it is not loaded in every device (camera was on the "main" dashboard). I am thinking of moving NodeRed to is own RPI. Is there a way to offload the WebRTC server to a separate device? |
@AngellusMortis I think it's better to find and fix the problem. There is some bug in code with certain cameras. I can't reproduce it with my cameras. |
is there any solution yet? |
If the problem cannot be repeated, it cannot be fixed. My raspberry 3b works without CPU issue with multiple cameras... |
It uses quite a lot of CPU on my Home Assistant Blue, especially when watching one that is 8 Mbit/s 1440p @25fps. I tried lowering its bitrate to 1 Mbit/s while keeping it on 1440p@25fps and that reduced the CPU use to about a third so it seems bitrate related. Would it be possible to run rtsp2webrtc on a different server instead of on the HA host? I already have a relatively powerful server that runs rtsp-simple-server and Frigate and many other things and wouldn't mind having a permanent rtsp2webrtc docker running on it if it could offload the Home Assistant host in some way. |
We have the same issue on our i3-8300. I disabled the integration in the configuration tab and the process was immediately killed, including the corresponding cpu load. I'd really love to use this integration, since it feels so much faster than other solutions. |
Same issue for me using UniFi Cameras. Both CPU & Ram was running very high. Not stable enough to use at the moment...Going back to UniFi Protect integration which has a 5-10 second delay..but very stable and minimal CPU/Ram usage. |
I have the same issue: NUC i3-6100U, 3 Reolink cameras using the Sub stream to WebRTC in Frigate cards. Having to reboot every other day as the process appears to gain resource without releasing it. Happens even faster with Background: True in the card. |
Hello, |
Hi all, I've run into the same issue. Running Home Assistant Core 2021.12.10 on Docker. Machine runs an i7-7567U with 16GB RAM, no resource limiting on the container. It initially worked quite well for about a week, and now it will rarely connect to the RTSP feed, and when it does it is quite slow to make the initial connection. Refreshing pages rarely seems to have any effect. I do have 100 ports configured (50000 - 50099), should I reduce that? I just re-installed the plugin and will come back here and update on how I progress, but over the last 5 minutes so far so good. I have suspicions that it may be getting caught in a rapid recursive loop somewhere. Is there any debug logging I can enable (even if it means switching a flag in a file somewhere)? I have no knowledge of go, but I'm happy to do what debugging / profiling I can (if I can figure it out). Of course anything I discover, I'll come back and report on. |
Hi,
I've resorted to a button on my Dashboard that restarts the WebRTC process.
I'm thinking of automating this so that if CPU hits 100% for > 10 seconds
that it kills and restarts WebRTC. Seems like a hammer for a nut though
really.
…On Wed, 9 Feb 2022 at 13:55, Michael Thompson ***@***.***> wrote:
Hi all,
I've run into the same issue. Running Home Assistant Core 2021.12.10 on
Docker. Machine runs an i7-7567U with 16GB RAM, no resource limiting on the
container.
3 x HikVision DT363-28.
WebRTC v2.2.0
It initially worked quite well for about a week, and now it will rarely
connect to the RTSP feed, and when it does it is quite slow to make the
initial connection. Refreshing pages rarely seems to have any effect. I do
have 100 ports configured (50000 - 50099), should I reduce that?
I just re-installed the plugin and will come back here and update on how I
progress, but over the last 5 minutes so far so good. I have suspicions
that it may be getting caught in a rapid recursive loop somewhere. Is there
any debug logging I can enable (even if it means switching a flag in a file
somewhere)?
I have no knowledge of go, but I'm happy to do what debugging / profiling
I can (if I can figure it out). Of course anything I discover, I'll come
back and report on.
—
Reply to this email directly, view it on GitHub
<#64 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARMPEBJUEGMXBZYYRM3PHBLU2JW33ANCNFSM45CJNKUQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you commented.Message ID:
***@***.***>
|
I just ran into this. This is on a HA Blue device, with 6 cameras. The ramp was gradual but persistent. In my case, it was only one specific instance of the binary server causing problems. This happened multiple times tonight for me, so I think this is a repeatable issue. If a debug version is released, I might be able to profile it? |
Seems RTSPtoWebRTC is deprecated in favor of RTSPtoWeb. https://github.com/deepch/RTSPtoWeb Opened #258 to track. |
I'm seeing this too. I'm only using one Eufy indoor pan and tilt camera (this model: https://www.amazon.com/dp/B0856W45VL). CPU usage on a Raspberry Pi 4 starts at ~3% CPU but over time it grows to ~50% CPU. The CPU usage is all coming from |
Started noticing this as well. Using rtsp simple server as a proxy to my cameras. I am running 10+ cameras and have noticed that when working normally it works with 30-40% on a two core VM. Previously I had 32 cores allocated to the VM, and it used all available resources on that as well. Perhaps some sessions were never closed properly and continually run? |
Same problem here, I'm running in a hyperv container, on a xeon v2 processor. All the webrtc streams are hooked up to crappy raspi and v380 cameras directly with rtsp urls, 6 cameras simultaneously on one lovelace panel. system CPU use goes to 100% randomly, it smoothly ramps up from 0% to 100%, in the last day once it took 20 minutes, the other time it took 2 hours, very smooth ramp-up. I can log onto the container and the offending process is "rtsp2webrtc_v5_amd64". The HA system seems usable while this is happening so I have no idea how long this has been going on. Probably a while, I remember catching it at least once months ago but not knowing why it was at 100%. Even the cameras seem to still be working with the CPU stuck at 100%, so its possible this problem is under-reported. One thing I can do is scale back the number of virtual cores so at least it impacts the host machine less, but this is pretty annoying, it impacts other virtual machines running on the same hyperv host as the cpu resources are shared. It also might explain why my HA system seems a bit laggy and unstable all the time. I really want to keep this integration though because its the only one that lets me PTZ my cameras when im not home without 20 seconds of lag. @AlexxIT we understand that you cant reproduce on a raspi but it seems that everyone with this problem is using x64 based platforms. Is there anything we can do to help investigate or fix this problem? We can all reproduce this it but we dont know how to help fix it. |
My current workaround is a root cronjob to kill it every few hours. Not ideal, but it helps:
I'll likely put a CPU limit on the Docker container it's running in, too.
I'm hitting the issue on a Raspberry Pi 4B running 32-bit Raspberry Pi OS |
I'm curious if this issue still occurs if rtsp2webrtc is ran directly. Since I'm not familiar with how exactly this component works, that would likely help to track down where the issue is. |
it happened again and I ran "top" and "ps aux | grep rtsp" been monitoring this for a few days now. doesnt seem to happen at night so far (when nobody looks at cameras). Does anyone else here use the android app or external access? we do all the time over here Note that for me, a normal cpu level is 2 to 5% even when watching the cams, so in the screenshots its off the charts high |
@timmeh87 I am using both the app and external access. Seems it's also working on internal access using the app. I can't confirm whether there is indeed a correlation between app usage and this issue however. |
Previously I was having the CPU go to 100% pretty much every day, then I SSH'd into my only raspberry-pi based camera and screwed around. My video recording stops around 7:18 that day then the CPU goes to 100% starting at 7:30. Not sure If I did something with the video server on the camera that screwed up webrtc... I then got distracted and that raspberry-pi based camera has been offline for a week. I did start doing ping tracing around my house with pingplotter, and realized I have pretty bad packet loss (up to 5% in an hour during the day) on two of my other cameras. I moved the router around and it got better, but still not perfect. Ever since I removed the pi camera, the cpu has not gone to 100%, except one little spike that is so narrow I cant actually attribute it to webrtc (maybe it was just HA doing something else with the cpu? I dont remember if I reset the plugin at that time, as I will always do every time I see I have high cpu, i did not automate it) I have a new router on order to replace the crappy cisco one that is giving me packet loss (all cameras on one router have a problem so its clearly the router, one camera is like, 2 feet away from it right now) The potentially flakey raspberry pi camera was running a year-old version of v4lrtspserver, I was trying to get a newer version onto it and bring it back online sometime. Ill keep updating if I see anything interesting |
I went into the pi camera this morning to rebuild v4l2rtspserver and I messed around a little. within about 5 minutes of the video recording (on my dvr) starting back up, my HA cpu started going to 100%. Im not necesarily saying that starting the pi camera makes it happen but now im almost 100% sure its something to do with that particular camera, maybe the old version of v4l2rtsp that I replaced or maybe its still a problem with the new version, Ill keep testing Is anyone else using a camera via v4l2rtspserver? |
@timmeh87 my primary server is Intel i3. I use a raspberry for tests. Both servers have no problem with CPU. Could be a problem with a specific camera or a specific router |
I just re-built my v4l2rtspserver software on the pi, I was using a random build from a year ago up to this week. This particular pi is actually running a standard pi camera along with a logitech USB webcam, which is handled with some kind of transcoding task to convert it to h264 from the camera's native format. So its perhaps a little bit more complex of a setup and if rebuilding didnt fix it but disabling it does, then ill have to probably disable the streams individually to narrow it down I also disabled the "background" flag on my cameras using this webrtc integration after trying it out for a few weeks. It seemed like the android app would try to maintain a session literally until you kill the app, so you'd open the app the next day and all the cameras would be in various states of error (maybe cause the network switched repeatedly when I was out) and Id have to kill the app on my phone for the webrtc to reconnect |
I'm going to go and say I don't think it's an issue with v4l2rtspserver. I'm using rtsp simple server as a proxy to actual IP cameras and am getting this issue. It's also unlikely to be an issue with the router as well as I'm sure most people have the equipment on the same layer 2 network. Either way, I'm using an OPNsense vm if it matters. |
I recommend everyone take a look at the NEW RTSPtoWebRTC Integration that comes WITH Home Assistant. The RTSPtoWebRTC integration was introduced in Home Assistant 2022.2. It works perfectly. There's a great video as well that walks you through the entire setup process. All my cameras are now without delay...and no CPU issues. |
The built-in component doesn't fallback when webrtc can't be established. |
I have had no issues in the last two weeks on 2022.4.4 |
Hi, same issue here on ARM64. I got 2 apartments with 2 independent hardware and independent networks. Both have HA Core 2022.5.4 and WebRTC 2.3.0. One apartment runs on Google Coral Dev Board Mini (low-power MTK A35 CPU) and the other on a RockPi4B (RK3399 Dual A72 + Quad A53). Both are streaming from 1 camera only (Wyze Cam v3 with RTSP firmware, 720p H264). My router is a MikroTik hEX S and WiFi is MikroTik CAP AC. The Coral board is connected by WLAN, the RockPi board by LAN. There are about 20 IoT gadgets in each apartment, most of them ZigBee bulbs and switches, not a heavy load. Upon boot, top shows CPU usage 7% for python and 5% for rtsp2webrtc_v5_aarch64. CPU usage remains same for about 1 hour, then it starts to spike to 200%. The RockPi4 (RK3399) starts to thermal throttle and becomes very slow. The Coral (MT8167S) overheats and shuts down completely. During this 1 hour browser is closed and nobody is accessing HA and nobody is watching watching the cam streams. @AlexxIT I'm happy to help debugging the issue, just let me know what you need. |
I got a new router and put my ISP's stupid Puma-7 based Hitron gateway into "bridge" mode, now I am using an older netgear nighthawk AX and a very old dlink AC router. Removed from the wireless system was an old cisco gateway being used as an AP, and the previously mentioned Hitron gateway. So I guess you could say my problem disappeared as soon as I stopped using crappy gateway combo devices for wifi. EDIT: I also remember I had updated my v4l2-rtspserver installation on my pi camera Its maybe still a bug that this happens but if you can stop it from happening then 🤷 |
Hi @timmeh87 I have everything cabled and was still running into this issue. Now I am not and made no changes since my initial comment. Not to say good quality wi-fi won't make it better - I suppose that if rtsp2webrtc is waiting on high latency packets maybe it may have high CPU, but I suggest that with no insight into the networking of this software. The question becomes, I suppose, how would we ensure that it's not multiple different problems presenting with the same symptoms? |
Yeah the presumed packet loss / v4l bugs were probably just a trigger for
the actual problem, which I am assuming is something to do with "video
transcoding" but otherwise I'm pretty ignorant to what goes on inside this
plugin. Video transcoding is complicated and I have been assuming the
problem with this plugin happens when some internal error or exception with
the video stream is not handled appropriately. But that is just a wild
theory. Then according to my theory, the crash could be triggered by
anything that gives off bad video data
We could poke at it with sticks all day but real progress might require it
to actually be "on the debugger" when this happens. I dont know what that
means for this plugin exactly but when I develop software there is a way
for me to step through the lines of code one by one and view the memory
contents
…On Tue, May 17, 2022 at 10:13 AM Michael Thompson ***@***.***> wrote:
Hi @timmeh87 <https://github.com/timmeh87> I have everything cabled and
was still running into this issue. Now I am not and made no changes since
my initial comment. Not to say good quality wi-fi won't make it better - I
suppose that if rtsp2webrtc is waiting on high latency packets maybe it may
have high CPU, but I suggest that with no insight into the networking of
this software.
The question becomes, I suppose, how would we ensure that it's not
multiple different problems presenting with the same symptoms?
—
Reply to this email directly, view it on GitHub
<#64 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHTCBFSTFMLY22NE7O665RLVKOSRLANCNFSM45CJNKUQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I have some doubts that it's a networking issues as I haven't seen this issue anywhere else, and I'm running multiple enterprise layer 3 switches along with an entire homelab infrastructure. |
Wouldn't be transcoding, as there is no transcoding performed. The stream is relayed verbatim. |
Hmm ok i dont know what to call it but it does *something* to the video
because the input is h264 and just by the delay and visual glitches I see
in the output, its not the same h.264 stream I can view in vlc directly
from the camera
…On Tue, May 17, 2022 at 11:41 AM Ryan Bray ***@***.***> wrote:
Yeah the presumed packet loss / v4l bugs were probably just a trigger for
the actual problem, which I am assuming is something to do with "video
transcoding" but otherwise I'm pretty ignorant to what goes on inside this
plugin. Video transcoding is complicated and I have been assuming the
problem with this plugin happens when some internal error or exception with
the video stream is not handled appropriately. But that is just a wild
theory. Then according to my theory, the crash could be triggered by
anything that gives off bad video data We could poke at it with sticks all
day but real progress might require it to actually be "on the debugger"
when this happens. I dont know what that means for this plugin exactly but
when I develop software there is a way for me to step through the lines of
code one by one and view the memory contents
… <#m_3109727243393126657_>
On Tue, May 17, 2022 at 10:13 AM Michael Thompson *@*.*> wrote: Hi
@timmeh87 <https://github.com/timmeh87> https://github.com/timmeh87
<https://github.com/timmeh87> I have everything cabled and was still
running into this issue. Now I am not and made no changes since my initial
comment. Not to say good quality wi-fi won't make it better - I suppose
that if rtsp2webrtc is waiting on high latency packets maybe it may have
high CPU, but I suggest that with no insight into the networking of this
software. The question becomes, I suppose, how would we ensure that it's
not multiple different problems presenting with the same symptoms? — Reply
to this email directly, view it on GitHub <#64 (comment)
<#64 (comment)>>, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AHTCBFSTFMLY22NE7O665RLVKOSRLANCNFSM45CJNKUQ
<https://github.com/notifications/unsubscribe-auth/AHTCBFSTFMLY22NE7O665RLVKOSRLANCNFSM45CJNKUQ>
. You are receiving this because you were mentioned.Message ID: @.*>
Wouldn't be transcoding, as there is no transcoding performed. The stream
is relayed verbatim.
—
Reply to this email directly, view it on GitHub
<#64 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHTCBFWKPU4TAIEVM2IAMKTVKO43VANCNFSM45CJNKUQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
probably lost frames.
…On Tue, May 17, 2022 at 10:55 AM timmeh87 ***@***.***> wrote:
Hmm ok i dont know what to call it but it does *something* to the video
because the input is h264 and just by the delay and visual glitches I see
in the output, its not the same h.264 stream I can view in vlc directly
from the camera
On Tue, May 17, 2022 at 11:41 AM Ryan Bray ***@***.***> wrote:
> Yeah the presumed packet loss / v4l bugs were probably just a trigger for
> the actual problem, which I am assuming is something to do with "video
> transcoding" but otherwise I'm pretty ignorant to what goes on inside
this
> plugin. Video transcoding is complicated and I have been assuming the
> problem with this plugin happens when some internal error or exception
with
> the video stream is not handled appropriately. But that is just a wild
> theory. Then according to my theory, the crash could be triggered by
> anything that gives off bad video data We could poke at it with sticks
all
> day but real progress might require it to actually be "on the debugger"
> when this happens. I dont know what that means for this plugin exactly
but
> when I develop software there is a way for me to step through the lines
of
> code one by one and view the memory contents
> … <#m_3109727243393126657_>
> On Tue, May 17, 2022 at 10:13 AM Michael Thompson *@*.*> wrote: Hi
> @timmeh87 <https://github.com/timmeh87> https://github.com/timmeh87
> <https://github.com/timmeh87> I have everything cabled and was still
> running into this issue. Now I am not and made no changes since my
initial
> comment. Not to say good quality wi-fi won't make it better - I suppose
> that if rtsp2webrtc is waiting on high latency packets maybe it may have
> high CPU, but I suggest that with no insight into the networking of this
> software. The question becomes, I suppose, how would we ensure that it's
> not multiple different problems presenting with the same symptoms? —
Reply
> to this email directly, view it on GitHub <#64 (comment)
> <#64 (comment)>>,
or
> unsubscribe
>
https://github.com/notifications/unsubscribe-auth/AHTCBFSTFMLY22NE7O665RLVKOSRLANCNFSM45CJNKUQ
> <
https://github.com/notifications/unsubscribe-auth/AHTCBFSTFMLY22NE7O665RLVKOSRLANCNFSM45CJNKUQ
>
> . You are receiving this because you were mentioned.Message ID: @.*>
>
> Wouldn't be transcoding, as there is no transcoding performed. The stream
> is relayed verbatim.
>
> —
> Reply to this email directly, view it on GitHub
> <#64 (comment)>,
or
> unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AHTCBFWKPU4TAIEVM2IAMKTVKO43VANCNFSM45CJNKUQ
>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#64 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAILPFEFDEP7YI7QBHPMCV3VKPFOHANCNFSM45CJNKUQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Check my new project. Maybe it handle this situation |
I am still seeing similar issues where it runs great but then starts taking over the CPU (running a HA Blue on ODroid). when the load starts causing my device to run slow I can disable and re-enable the integration to get things going great again. also trying out go2rtc, |
Should be fixed in v3 |
Hi,
Since I use the addon, I'm observed a masive and increasing cpu using. It starts at one determined time each morning and keeps rising until I restart home assistant. Then all be fine until next day.
Here you can see rtsp2webrtc_v4_ cpu use just before restart.
Do you have any idea about this?
The text was updated successfully, but these errors were encountered: