After a certain time, Supervisor Communication from Core seems to stop working

[Tscherno]

Describe the issue you are experiencing

Hi,

i'm using HASS OS as OVA on a Virtual Machine on my Synology. It worked for a few years without issues. Lately it started to have some very strange issue. It could be connected to the latest supervisor update, but i'm not 100% sure.

What it does:

• After Restart (OS or Core) everything works perfectly
• Some time later (probably 30 minutes) the problems begin

• Addons are not accessible via ingress anymore: trying to access them gives a timeout or Unable to load the panel source: /api/hassio/app/entrypoint.js.
• Addons itself are running (i see it for example for fully working Zigbee2MQTT addon in MQTT)
• Trying to access Settings => Addons gives "Could not load the Supervisor panel!"
• Restarting from the interface doesn't work anymore

The observer-statuspage gives

Home Assistant observer

Supervisor:	Connected
Supported:	Supported
Healthy:	Healthy

Restarting / Reparing the observer doesn't help

Restarting HA Core via CLI fixes the issue temporarly again.

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

1. Restart HA Core
2. Wait some time

Anything in the Supervisor logs that might be useful for us?

No noticeable errors

System Health information

System Information

version	core-2023.1.0
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
user	root
virtualenv	false
python_version	3.10.7
os_name	Linux
os_version	5.15.80
arch	x86_64
timezone	Europe/Berlin
config_dir	/config

Home Assistant Community Store

GitHub API	ok
GitHub Content	ok
GitHub Web	ok
GitHub API Calls Remaining	5000
Installed Version	1.29.0
Stage	running
Available Repositories	1208
Downloaded Repositories	46

Home Assistant Cloud

logged_in	true
subscription_expiration	November 30, 2023 at 1:00 AM
relayer_connected	true
remote_enabled	true
remote_connected	true
alexa_enabled	true
google_enabled	false
remote_server	eu-central-1-4.ui.nabu.casa
can_reach_cert_server	ok
can_reach_cloud_auth	ok
can_reach_cloud	ok

Home Assistant Supervisor

host_os	Home Assistant OS 9.4
update_channel	beta
supervisor_version	supervisor-2022.12.1
agent_version	1.4.1
docker_version	20.10.19
disk_total	491.4 GB
disk_used	56.8 GB
healthy	true
supported	true
board	ova
supervisor_api	ok
version_api	ok
installed_addons	Samba share (10.0.0), File editor (5.4.2), SSH & Web Terminal (13.0.0), Grafana (8.1.0), MariaDB (2.5.1), Frigate (0.11.1), Node-RED (14.0.1), evcc (0.110.1), Eufy Security Add-on (1.3.0), UniFi Network Application (2.5.0), phpMyAdmin (0.8.3), ESPHome (dev) (dev), RTSP Simple Server Add-on (v0.17.6), Zigbee2MQTT (1.29.0-1)

Dashboards

dashboards	4
resources	32
views	9
mode	storage

Miele

component_version	0.1.4
reach_miele_cloud	ok

Recorder

oldest_recorder_run	December 30, 2022 at 8:56 AM
current_recorder_run	January 6, 2023 at 9:47 AM
estimated_db_size	1002.31 MiB
database_engine	mysql
database_version	10.6.8

Sonoff

version	3.3.1 (b20e33c)
cloud_online	0 / 1
local_online	0 / 0

Supervisor diagnostics

config_entry-hassio-a0598a45fdd52b1dd951477d67f04d27.json.txt

Additional information

No response

[thomasheiser85]

Have the same issue. I'm using HASS OS as OVA on a Virtual Machine on my QNAP

[Tscherno]

Have the same issue. I'm using HASS OS as OVA on a Virtual Machine on my QNAP

Same environment for me.

[maikelvds]

Also here, odroid n2+

[VeZReVouLiS]

Same issue and in SU logs i have the following :
"23-01-06 20:46:34 ERROR (MainThread) [asyncio] Task exception was never retrieved
future: <Task finished name='Task-2526' coro=<Addon.watchdog_container() done, defined at /usr/src/supervisor/supervisor/addons/addon.py:988> exception=AddonsJobError('Rate limit exceeded, more then 10 calls in 0:30:00')>"

For the moment i disabled the watchdog addon and the issue has stopped (not sure if it is a random or not, time will tell)

Running HAOS on VM (Virtual Box)

[francois-nitenberg]

I'm having exactly the same behaviour. It seems to have started after upgrading to 2023.1.0/1 but still occurs after reverting to previous snapshot.

[maikelvds]

I had it after 2022.12.9.
Seems to be a problem in my 'Home Connect' data in the configurator.
Deleted that part and till now no issues anymore

[teknology]

I'm having the same issue using HAOS on a mini pc. Everything was working great with the mini pc until about 2 days ago. Looking at CLI it's printing out some weird error repeatedly.

I hope this gets resolved soon otherwise I may as well cancel my cloud subscription.

[maikelvds]

I had it after 2022.12.9. Seems to be a problem in my 'Home Connect' data in the configurator. Deleted that part and till now no issues anymore

Problems are back again..

[VeZReVouLiS]

For the moment i disabled the watchdog addon and the issue has stopped (not sure if it is a random or not, time will tell)

Running HAOS on VM (Virtual Box)

Same for me.
The issue came back.

[patrick131184]

same here

[francois-nitenberg]

Issue still occurring for me too. I'm running the full HA OS on a x86 PC.

When I try accessing any add-ons from the left hand-side navigation menu, I get the following error:
Unable to load the panel source: /api/hassio/app/entrypoint.js.

However, this health-check page doesn't report any problem:

From the logs, there is a lot of noise from addons and integrations but I believe those are the consequence and not the root cause. The only suspicious trace I can see is:

Logger: homeassistant.components.hassio
Source: components/hassio/__init__.py:488 
Integration: Home Assistant Supervisor (documentation, issues) 
First occurred: 6 January 2023, 19:13:45 (219 occurrences) 
Last logged: 14:03:43

Can't read Supervisor data:

[maikelvds]

Do you guys have a integration which is not working well?
Yesterday I fixed my Eufy integration. Till now everything is working fine. I also went back to a bsckup from before the Problems started. After this I restored the latest backup, so no data loss.

[Tscherno]

I just tried to disable my Eufy Integration. Let's see if this make a difference. But even so, a problemactic Addon shouldn't have such an effect, shouldn't it?

[patrick131184]

using eufy also, and have some issues with it. disabled now, lets see if it makes a difference

[jgracio]

Same problem here with eufy now disabled. I was going mad with my HA, but thanks to the community and @Tscherno first report, my system seems to be back on track.

I believe Supervisor have to be more robust and not get so much affected by bad behaviour integration.
.

[thomasheiser85]

I also had a not working Eufy Addon. After entering a TRUSTED_DEVICE_NAME (any name) in Eufy Addon config the integration worked fine again.

Till now the problem seems to be fixed for me.

[maikelvds]

I also had a not working Eufy Addon. After entering a TRUSTED_DEVICE_NAME (any name) in Eufy Addon config the integration worked fine again.

Till now the problem seems to be fixed for me.

That's also what I fixed. Almost 24h without losing the supervisor. So I think that was the problem. It is strange, for sure.

[VeZReVouLiS]

I just made it for myself in eufy settings as well.
Now it remains to be tested...

[basje56]

Ran in to the same problem as described. Adding the TRUSTED_DEVICE_NAME in the Eufy Addon fixed it form me i think. Currently 24 hours without Supervisor loosing connection. Thanks to this thread!

[supergroover]

I just updated the google drive backup addon and have the same problem .

[sforeman00]

I disabled Eufy just to see if it would stop losing the supervisor and it worked. I will be adding the TRUSTED_DEVICE_NAME to see if i can re-enable Eufy.

[mdegat01]

Can someone link the Eufy addon you all seem to be using? Have never heard of it. And very confused how an addon could break internet connection.

[maikelvds]

It it the integration out of HACS.
Link: https://github.com/fuatakgun/eufy_security/

Since the fix I don't have problems anymore.

[mdegat01]

I see, this has both an integration and an addon. Tbh I'm not sure how either of these could break supervisor in the way you all are describing. And since Observer seems to be reporting supervisor is alive and healthy I think its actually core that is being hosed by the integration. A custom integration can absolutely bring down or completely core if behaving badly since they run right in the same python process as the rest of HA.

A good way to confirm my suspicion would be to test some communication mechanisms that go around core. Next time someone on here gets into the situation being described can you try a simple test for me? SSH in and run some commands in the CLI like:

ha supervisor info
ha addons info

Could also try restarting an addon (ha addons restart <addon slug>). Just simple tasks to confirm supervisor is working.

Note - you must SSH in or use the host console (connect a monitor and keyboard) for these tests. Don't use the SSH panel in the UI. That panel only works if both core and supervisor are both working so it invalidates the test since we're trying to see if core or supervisor is frozen.

[mdegat01]

Also for folks on this thread I want to narrow down the issue. Can someone try disabling the integration (not the addon, just the integration, leave the addon running) and let me know if that fixes the issue. Then we know exactly what the source of the problem is.

[q1t1]

Hi,

Same problem here, but no Eufy integration installed.

Edit :

Again this morning, with the following logs :

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:476
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 07:01:12 (22 occurrences)
Last logged: 07:44:19

Timeout on /info request
Timeout on /ingress/session request
Timeout on /network/info request
Timeout on /os/info request
Timeout on /host/info request

---------------------------------------------------------------

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:48
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 06:06:38 (160 occurrences)
Last logged: 07:44:23

Timeout on /core/info request
Timeout on /info request
Timeout on /supervisor/info request
Timeout on /host/info request
Timeout on /store request

---------------------------------------------------------------

Logger: homeassistant.components.hassio
Source: components/hassio/__init__.py:488
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 06:07:45 (9 occurrences)
Last logged: 07:44:23

Can't read Supervisor data:

The commands "ha addons start/stop/restart " work so it's a core problem (connection between core and supervisor), not a supervisor one.

[q1t1]

Hi guys,

Any updates on your sides ?

[Pop70]

Same problem. HasOs raspberrypi 3
config_entry-hassio-9b5808b5a554f49cf90795820566f97f.json.txt

Logger: homeassistant.components.homeassistant_alerts
Source: helpers/update_coordinator.py:168
Integration: Home Assistant Alerts ([documentation](https://www.home-assistant.io/integrations/homeassistant_alerts), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+homeassistant_alerts%22))
First occurred: 13:18:14 (1 occurrences)
Last logged: 13:18:14

Timeout fetching homeassistant_alerts data

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:48
Integration: Home Assistant Supervisor ([documentation](https://www.home-assistant.io/integrations/hassio), [issues](https://github.com/home-assistant/home-assistant/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+hassio%22))
First occurred: 22 января 2023 г., 00:27:01 (186 occurrences)
Last logged: 14:51:26

Timeout on /os/info request
Timeout on /supervisor/info request
Timeout on /core/info request
Timeout on /store request
Timeout on /info request

[Tscherno]

I was able to fix my issues by updating the Eufy integration and put the parameter in. My guess it, that it was in some restart loop and the watchdog went crazy. I cannot reproduce it anymore.

[q1t1]

Just to keep this open, still have the issue...

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[B0ndo2]

I have the same issue and I don't have Eufy.
Supervisor 2023.03.01
It is happening now frequently now after updating to this version

[ghost]

I have a very simple installation with Hass OS (which does not include Eufy) on a Pi 3B and have very similar symptoms to those described.
Hardware and PSU check out ok. A fresh install of HA followed by reloading a backup has not resolved the issue. For me the problem seemed to appear after updating to 2022.12. Previously HA has run flawlessly in the environment for over 5 years. Now searching for an older backup to try.

[B0ndo2]

Any help please. The system doesn't run for more than a day before this happens.
I updated to the recent version and still no use

[q1t1]

Hi,

Please check your system logs (search for recurring errors).

For me, it was a problem with an android box (set with Android TV integration) that was sleeping and unavailable and generated hundreds of errors.

Regards,

[B0ndo2]

Hi,

Please check your system logs (search for recurring errors).

For me, it was a problem with an android box (set with Android TV integration) that was sleeping and unavailable and generated hundreds of errors.

Regards,

I haven't added or changed anything, just updated the core.
Which log should I look at?

[q1t1]

Hi,
Please check your system logs (search for recurring errors).
For me, it was a problem with an android box (set with Android TV integration) that was sleeping and unavailable and generated hundreds of errors.
Regards,

I haven't added or changed anything, just updated the core. Which log should I look at?

Settings > System > Logs

[B0ndo2]

Hi,
Please check your system logs (search for recurring errors).
For me, it was a problem with an android box (set with Android TV integration) that was sleeping and unavailable and generated hundreds of errors.
Regards,

I haven't added or changed anything, just updated the core. Which log should I look at?

Settings > System > Logs

There are core logs, supervisor logs, etc..

[q1t1]

There are core logs, supervisor logs, etc..

Core

[Tscherno]

Any help please. The system doesn't run for more than a day before this happens. I updated to the recent version and still no use

I wasn't able to tell from my logs. Try to disable all add-ons first and enable one at a time only. See if the problem still exists. Hopefully this shows which one causes the issue. If no success, try the same with device integrations.

[B0ndo2]

The problem seems to be coming from Frigate addon but what doesn't make sense is how can an "addon" affect the whole system like this . Disabling the addon is not a solution.

[escomputers]

Same problem here, version 2023.3.5 core on raspberrypi4-64
config_entry-hassio-cd2d793cc663b2331d864c3fd3de1012.json.txt

No Eufy addon, my addons were:

• cloudflare
• samba
• file editor
• terminal & ssh
• vs code server

I've also tried to disable all integrations but still gettting:
Unable to load the panel source: /api/hassio/app/entrypoint.js.

If I run ha addons info I get this error:
Get "http://supervisor/addons/self/info": dial tcp 172.30.32.2:80: connect: no route to host

The observer at ip:4357 says:
Supervisor: Disconnected

Home Assistant Core errors list:

Logger: homeassistant.components.hassio
Source: components/hassio/__init__.py:487
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 23 marzo 2023 alle ore 22:53:34 (2751 occurrences)
Last logged: 15:21:26

Can't read Supervisor data:

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:48
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 23 marzo 2023 alle ore 22:53:34 (27510 occurrences)
Last logged: 15:21:26

Client error on /host/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /store request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /core/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /supervisor/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /os/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]

Logger: homeassistant.components.hassio.http
Source: components/hassio/http.py:181
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 24 marzo 2023 alle ore 14:30:52 (58 occurrences)
Last logged: 15:17:12

Client error on api app/entrypoint.js request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on api supervisor/logs request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on api host/logs request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]

Logger: homeassistant.components.hassio
Source: components/hassio/websocket_api.py:125
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 22 marzo 2023 alle ore 23:33:09 (96 occurrences)
Last logged: 15:13:57

Failed to to call /backups -
Failed to to call /host/info -
Failed to to call /supervisor/info -
Failed to to call /network/info -
Failed to to call /addons -

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:35
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 24 marzo 2023 alle ore 14:31:45 (9 occurrences)
Last logged: 15:14:01

Client error on /homeassistant/restart request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]

Logger: homeassistant.components.hassio.handler
Source: components/hassio/handler.py:482
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 24 marzo 2023 alle ore 14:31:38 (100 occurrences)
Last logged: 15:13:57

Client error on /host/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /backups request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /addons request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /supervisor/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
Client error on /addons/samba/stop request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]

[B0ndo2]

In my case it was Frigate. I disabled it and everything was fine. I don't see a near term solution so right now am restarting at 5 AM using an automation

[escomputers]

is there a way to disable/remove addons without UI?

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[tader]

I think the issue was not solved... Don't close it yet.

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.