Permanent error: Client exceeded max pending messages [2]: 512 #26724
Comments
|
I have the same stupid issue. Hass is unusable. As stated in op there is no clear reason for this error. Examples by other posts indicate the cause is completely unknowable at this time. Maybe try NOT closing the issue again all? You're decentralizing the attempts to troubleshoot with no possible gain in doing so. |
|
Getting same error |
|
Updated to 101.2 - HA becomes unusable at all. I have hundreds of such errors every minutes. All server CPU wastes to error log write. |
I updated, same result. I'm leaning toward starting from scratch. Such a hassle. |
|
I'm getting same error from fresh hass.io HA 101.3 install. Also tested plain 101.3 HA venv install without hass.io and that one works without generating errors to log, so my guess is that problem is related to hass.io docker container permissions/routing. |
|
Same issue with 0.100.2. |
|
Same error with 0.101.3 (in a virtual environment)... I started to track that in the logs ... Here is the last extract from the logs (HA seems to work fine despite this recurring error message): |
|
Same here: It seems to be happening sometime after I'm connecting to HA from the Android webapp. |
|
Found what was causing these errors for me and can reproduce 100%
Standard browser refresh clears the error loop. So I guess I just have to remember to refresh my browser after editing a card. Possibly a polymer/frontend issue with how HA attempts to refresh Firefox. |
|
I still have this error message in 0.103.5, but more rarely yet. |
|
Issue in frontend part: home-assistant/frontend#5328 |
|
On 107.7 HA is very unstable. Interface periodically becomes unavailable and CPU usage of HA server increases from 5% to more than 40%. During such a period I see a tons of errors: |
|
The problem is still there in 0.108.2... After a request from lovelace (via a button) or through the development tools to reload automations, 102 message in the same second "2020-04-11 05:49:36 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection.2614746480] Client exceeded max pending messages [2]: 512" ... |
|
I'm seeing this issue as well. I noticed that removing every card that used card-templater seemed to resolve the issue for me but that's obviously not an ideal resolution... |
|
Same in 0.108.8 - Happens randomly, have to restart home assistant to get it work again. Solved the continuous errors, here the steps from my post in community.home-assistant.io: After many tries I've found that the "problem" is hacs and/or card-tools... I have to find if directly related to that integrations or if a sum of other issues... Let me explain:
During the clean i've found a lot of unused old stuff and some js that where initially installed manually and then via hacs... So probably the issue is from that confusion beside the custom_components itself... Sometimes the issue re-happen, Hass get's stuck for some minutes and then runs correctly. |
|
109 includes a fix to allow websocket API clients to catch up for 5 seconds if they fall behind on consuming messages. |
|
I think is related to chrome tabs that remains open for long time. More details below:
Probably Hass and Frontend does not manage correctly the socket disconnection-reconnection. Hass must stop sending data through websocket after first pending messages and meanwhile Frontend must attempt a reconnection (like when hass is restarted but ui is keeped open) Hope it helps |
|
I fully agree with your part of analysis saying, that it's always caused by clients having UI open for long time. |
|
I have a similar issue. It's getting frozen, my automation are really slow, etc. |
|
Check your logs size, it's getting frozen beacuse of gigabytes it has to write to disk ;-) |
|
I don't think so, the logfile is "only" 100MB, with more than 550'000 error lines (and it's working for the time being). |
|
With more than 800 entity... can You imagine the traffic on the socket... For me everything starts happening last week, I've updated hass but didn't remember previous version (last working). No more entities added, the only update was to home assistant... Another fact is that with the number in logs I can't identify which device causes the issue, maybe in the error message a client identifier (much more detailed) can be usefull... |
|
Just found that the client is causing all is appdaemon.... Didn't know if itself or specific app Update: AppDaemon is the client that shows up on logs, isn't clear if is the cause or a symptom |
|
I've also been seeing this occasionally on 0.108 and 0.109 - Does seem to affect appdaemon, as my appdaemon automations all stopped working. However, I still saw the errors in the logs continuing after I stopped AppDaemon. Some combination of closing HA tabs in my browser and an HA restart finally cleared it up (for now). |
|
Also in 109.1, with appdaemon 4.0.3 At 22:10 I've received the message from the watchdog... Hass was unresponsive and doesn't make any automation (as the watchdog is based on automated mqtt from hass) logs from appdaemon: didn't know how to do at this point |
|
Same problem here. I noticed it started happening when editing the lovelace interface and i had just started using layout-card and card-mod with button-card. As of 0.109 the issues got worse: I dont get the "Connection Lost" popup anymore but the interface is slow, sluggish, sometimes fails to load and the page i was working on is now broken: |
|
Its going to depend on the performance of the system, latency of the connection, and how fast the client can process the messages. On my system, it can easily process 100 state changes event per second. On an RPi3 it will fall over. 30 every 5 seconds is a bit high if they all get delivered at the same time. Most, but not all systems will be able to keep up. I'd expected the lower end hardware will fail at that level. Using |
|
@bdraco Thanks! This helps a lot. Having it as an option for the integration is also not really ideal, looking at this specific integration only maybe 5 out of the 30 entities are really relevant for Perhaps maybe somewhere in future this can be added to customization, to allow users to set it entity specific... |
|
just wondering if anybody else is finding this issue related to automations where you are running update_entity on a sensor. I had a few, in different automations, in HA .118.3 the error message 512 for 5 seconds is always appearing right after an automation is running. When I comment out update_entity and reload automation the cpu usage has dropped down as well at least 10%. |
|
I have the same issue, started after update 0.118.5 running hassos 4.17 on RPI 4gb Running hacs integration, ipp and flux led and limitless led I get loads of update of light taking longer than 10 seconds would love to see some actual solutions on these issues. Having said that, I love HA and it’s enormous flexibility. |
|
I used to get these messages pretty regularly on an RPi 3B+. Several dozen a day. My experience was that mostly the messages can be ignored, but some integrations generate so many state changes it triggers these errors to the point that Lovelace gets so laggy that it's unusable. A few different things would trigger this, early versions of the EdgeOS integration, heavy use of auto-entities, heavy use of card-mod to change colours of icons based on state. I figured my HA install was just outgrowing the Pi. However a few weeks ago I moved my HA install to VirtualBox on an i7 Macbook Pro, and these messages are reduced but I still get them. Maybe around 5 a day now. |
|
I've reduced my overall cpu usage on the RPi 3B+ from 40% to barely 20% (average now is only 15%), I still see a few of these messages "512 for 5 seconds" every day. When one of my integrations running in the homeassistant docker container stopped for 1 week by me (hardware died so I stopped automations), the message went away completely. It makes me speculate either a HA resource is synchronized and all integrations have to take turns to call the shared api, or its docker restricting cpu usage on each container. My overall RPi 3B+ is only 20% max at the raspbian buster level, but when I run "docker stats" is see the homeassistant container reach 150% cpu, but the RPi 3B+ still has plenty cpu power. I'll have to try find some docker command to see if I can give more CPU resource from the Hardware to the homeassistant container only, its the only container that is consistently cpu hungry. I run homeassistant in supervised installation, so I may have some flexibility to try these things, but not 100% sure. |
|
this might help someone else, I've narrowed this down to too many history graphs. my guess each graph plotting sensor data, is querying the mariadb at the same time. When I moved my graphs from one raspberry pi to another to offload some the overhead, the pending messages moved to the new raspberry pi with the custom sensor history graphs I have setup. |
|
@JIOB Have you fixed this problem? I'm also using full kiosk browser on my tablet and having the same issues. |
@ceesvanegmond I managed to track my issue to a script I have to automatically switch my FullyKiosk instances on/off. I increased the delay on switching the wall tablet off and the problem went away. |
|
@JIOB Are you be able to share that script? Very interesting! |
Sure, the script essentially keeps my wall tablet on until no motion is detected. I increased the delay in 'hallway_panel_timer_off' from 1 second to 5 and the problem went away. As for the script I'm sure there are better ways of doing this, however, I have ran this script and others like it for a long long time and they work for me. |
|
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. |
|
Still occurring when backgrounding the iOS app. |
|
Also still experiencing this. |
|
Me to. this message has started appearing in my logs, no further detail so assume I will have to turn up the detail on logging. Running HASSOS 2021.5.5 |
|
Sane here: Logger: homeassistant.components.websocket_api.http.connection [547571963792] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds |
|
Happening here as well. I do not have AppDaemon, definitely seems to be front-end related. Not sure if possible for somewhere to add some more info when this occurs to better get down to what is causing this? Seems that it might not always be 1 thing and is something that can come back from time to time. |
|
There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. |
|
This is still happening for me on core-2021.11.1. |
|
For me, too |
|
I have the same issue on 2021.11.5 |
|
I may have a suggestion, I can't prove it'll work for anybody, and if my idea still requires a tweak. I was getting "Client exceeded max pending messages" because I have too many history graphs. Graphs have to query the database for entity_id, state, last_updated. I'm using mariadb so I used a tool DBEAVER to connect to HA, I took a peek at the index in homeassistant "states" table, the closest index I found that could be used for history graphs is named "ix_states_entity_id_last_updated" it has columns (entity_id, last_updated) only. so a SQL query used to retrieve the data to draw history graphs would have to use the index then read the table by rowid to retrieve state data, to plot column state on y axis, column last_updated on x axis for a specific entity_id. My hunch for my case this is where the bottleneck is for my hardware rbi3b+ with a solid state drive. as an experiment I've added a new Index "ix_states_entity_id_last_updated2" to help optimize the query just slightly so it wouldn't have to access the table to get the needed data, the index storage size will go up slightly, but everything the query needs will be in the index and less disk I/O here's the DDL to add the index for mariadb, if you don't think the index can be unique for the 3 column combination then remove the UNIQUE keyword. here's the DDL to drop the index for mariadb, after you're done with the experiment I'm still looking at my logs for the max pending messages, I haven't seen it lately, that could be because of my experiment or HA version 2021.11.5 helped on the solid state drive hardware I use. if the experiment is successful, than the index "ix_states_entity_id_last_updated" would be replaced with the change, we shoudn't add another index unless there is a need, the rbpi3b+ has only so much ram to cache indexes. the above DDL would be changed to, that is if mariadb allows create or replace from an index that is not unique to an index that is unique. if it doesn't you would have to drop the index 1st, then add it. |
|
@pierre2113 Any updates on your experiment? |
|
I'd forgotten about this error, I had other HA fires I had to fight. But looking at my logs I don't see it. |
and others
Home Assistant release with the issue:
HA 0.98.5
Last working Home Assistant release (if known):
HA 0.91 (0.92.0?)
Operating environment (Hass.io/Docker/Windows/etc.):
HASS.io/Docker/Raspbian Buster
Component/platform:
homeassistant.components.websocket_api.http.connection
Description of problem:
I have to reopen the issue (#23938) since it was unreasonably closed (some people have the same problem but in different environment).
To be more exact, no use of Node-Red and no HA automation restarts but issue is till on.
Problem-relevant
configuration.yamlentries and (fill out even if it seems unimportant):Tons of errors in error log with no any clear reason. I can have more than 15.000 such errors a day.Traceback (if applicable):
Additional information:
The text was updated successfully, but these errors were encountered: