Client unable to keep up with pending messages #90281
Comments
|
Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration ( Code owner commandsCode owners of
(message by CodeOwnersMention) websocket_api documentation |
|
Update to 2023.3.x. It should give you more information about why |
|
Also please post a py-spy using version 0.3.14 https://community.home-assistant.io/t/instructions-to-install-py-spy-on-haos/480473 And a callgrind.out file from the profiler integrations profiler.start service |
|
Are you using Node Red or AppDaemon? |
|
Yes i am using Node Red. ps. but i am using nodered only to play one automation: sending mail with attachment. Only one automation and its all |
|
Try disabling the node red custom component |
|
i uninstall nodered and problem hit today three times. @bdraco below file callgrind.out i just start service profiler and this file generated:
event: ns : Nanoseconds
events: ns
summary: 1001499731
fl=/config/custom_components/ble_monitor/binary_sensor.py
fn=async_run
108 34483
cfl=/usr/local/lib/python3.10/_weakrefset.py
cfn=_remove
calls=1 39
108 5015
cfl=/usr/local/lib/python3.10/asyncio/tasks.py
cfn=wait_for
calls=3 392
108 171216
cfl=/usr/src/homeassistant/homeassistant/util/dt.py
cfn=now
calls=1 108
108 9541
cfl=~
cfn=
calls=1 0
108 601
|
|
Please post the callgrind file as an attachment |
|
callgrind.out.1682337757952494.txt i added .txt because this file format isnt suported here |
|
|
|
Something is firing very large events and its taking a long time to serialize them. Please enable debug logging for |
|
ok enabled |
|
Look for large events in the log |
|
i enabled it about 5:50 pm and restarted HA. |
|
Please look at this stats below. This is my cpu and ram load. this state is maintained practically all the time, there are no spikes, the system is never loaded. Increased load is only when backup is made but it is 1x a day and it takes maybe 30 seconds.
|
This comment was marked as abuse.
This comment was marked as abuse.
|
If an integration has a bug that blocks the event loop, cpu and ram is irrelevant and you would only see it in the callgrind or by turning on async IO debug |
|
Short of reducing whats going into the queue, we could improve the queue throughput by making the size unbounded since we only have one place items come into the queue, and we could check it there. That would turn off all the internal size checks where are a bit of a bottleneck to adding things quickly to the queue. But I'm not sure if thats going to help in this case |
|
We really don't need a full |
|
I put #92967 on my production. I ran |
|
Perhaps I should create a different PR, but starting 2023.6.0/2023.6.1, I'm seeing this error as well. It seems I only get this error when a persistent notification is sent (showing up in the "Notifications" sidebar). I also have a problem with the Android/Lenovo Tablet, whereby at the UI, when I use the sidebar Notifications to clear the notifications, they are cleared/dismissed, but the sidebar continues to indicate I still have a notification (circle with number of notifications pending). This maybe because of this error. I don't have this issue with the other browsers/companion app. |
|
You probably have something that is sending the notification over and over and over again. |
|
I did the following: then hit "Call Service" and immediately the same Log showed up. |
|
I'm seeing this error on 2023.6.1 as well. I'm having difficulty reproducing it reliably, but it appears to be triggered by notifications, as indicated by others. The companion app gets into a loop of continually reconnecting and then disconnecting immediately. |
|
Same problem here. The device referred to is an iPad Mini 5. Home Assistant 2023.6.2 And |
|
Same here, iOS App needs to get restarted all the time :-/ |
|
While its a generic error and we can never fix every instance of it on every device. These two PRs will improve the situation for mobile / narrow devices: |
I have the same behaviour. |
|
I'm having the same problem. More or less since upgrade to 2023.6.0 it started on my mobile phone and today, after upgrading to 2023.6.3 also on a Chrome browser running on Windows 11 (powerfull machine). Here is resource utilization of HA box (Debian running HA container). Ignore the peak to 100% as esphome was upgrading all my devices at that time: The last one occurs over 700 times right after it. |
bdraco
changed the title
bdraco
changed the title
|
Thanks, I'll have a look. One point only: I've disabled automations that were creating persistent notifications and now I don't face this issue anymore. But don't get persistent notifications neither. Also as mentioned, this was not happening before one of the recent HA Core upgrades and I have not been adding any new integrations recently. |
|
Its impossible to tell for sure if its fixed by the linked issues above without the callgrind and py-spys |
|
Understood. Yet already a few hours since I've disabled all persistent notifications and the issue has not returned neither on mobile nor on PC chrome browser. I'll need some more time to collect the callgrind and py-spys. |
|
2023.7.0: Still seeing these. |
|
Yes still there. I have disabled all my persistent notification in automation but sometimes I am getting notification about failed login and pressing dismiss button is crashing HA. |
|
Is it still an issue in core-2023.7.1 for you people? Because for me the issue has gone. (fingers crossed). |
|
I still have it. |
|
I've tested it a couple of times today on 2023.7.1, and this issue did not show up :) |
|
Issue is gone since 7.0 here - thanks guys! |
|
7.1 I think issue is gone. |
|
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. |
issue-triage-workflows
bot
closed this as not planned
Read before posting
#95386
The problem
this error at log.
Systemwasnt load too high
cpu all the time about 15-25% and memory 30-40%
What version of Home Assistant Core has the issue?
2023.2.5 and newer
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant OS
Integration causing the issue
Home Assistant WebSocket API
Link to integration documentation on our website
https://www.home-assistant.io/integrations/websocket_api
Diagnostics information
No response
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
No response
The text was updated successfully, but these errors were encountered: