A lot of connection issues for a lot of integrations: SSL/443 - helpers/update_coordinator.py:350

[Phoenix-DH]

The problem

Hello together,

to be honest, I am not pretty sure when the issue started, but for me it seems to be with 2024.3.
At home I changed nothing with my network connections/settings/components.
I have also since a longer period of time SSL active for my instance to get SmartThings and web access working.

At my wall I have a dashboard where I regulary see my stats.
I can see that at the point of time where my components/integrations get not working, I still have network and everything up and running.

Affected are for example openweathermap/apple icloud/google calendar ....

Attached are some logs.

I have absolutely no clue what can be the cause for that.

Do others also have this issues?

What version of Home Assistant Core has the issue?

2024.3.1

What was the last working version of Home Assistant Core?

2024.2.X

What type of installation are you running?

Home Assistant OS

Integration causing the issue

openweathermap/apple icloud/google calendar

Link to integration documentation on our website

No response

Diagnostics information

Logger: homeassistant.components.openweathermap.weather_update_coordinator
Source: helpers/update_coordinator.py:350
integration: OpenWeatherMap (documentation, issues)
First occurred: 2:10:00 AM (7 occurrences)
Last logged: 7:51:29 AM

Error fetching openweathermap data: HTTPSConnectionPool(host='api.openweathermap.org', port=443): Max retries exceeded with url: /data/2.5/weather?APPID=6ef58e19b43b7a891b500b38c1976701&lang=de&lon=7.808259129524232&lat=49.15410290594424 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5f5096a0>: Failed to establish a new connection: [Errno -3] Try again'))
Error fetching openweathermap data: HTTPSConnectionPool(host='api.openweathermap.org', port=443): Max retries exceeded with url: /data/2.5/weather?APPID=6ef58e19b43b7a891b500b38c1976701&lang=de&lon=7.808259129524232&lat=49.15410290594424 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5e9133b0>: Failed to establish a new connection: [Errno -3] Try again'))
Error fetching openweathermap data: HTTPSConnectionPool(host='api.openweathermap.org', port=443): Max retries exceeded with url: /data/2.5/weather?APPID=6ef58e19b43b7a891b500b38c1976701&lang=de&lon=7.808259129524232&lat=49.15410290594424 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f60d28680>: Failed to establish a new connection: [Errno -3] Try again'))
Error fetching openweathermap data: HTTPSConnectionPool(host='api.openweathermap.org', port=443): Max retries exceeded with url: /data/2.5/weather?APPID=6ef58e19b43b7a891b500b38c1976701&lang=de&lon=7.808259129524232&lat=49.15410290594424 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5f5623f0>: Failed to establish a new connection: [Errno -3] Try again'))
Error fetching openweathermap data: HTTPSConnectionPool(host='api.openweathermap.org', port=443): Max retries exceeded with url: /data/2.5/weather?APPID=6ef58e19b43b7a891b500b38c1976701&lang=de&lon=7.808259129524232&lat=49.15410290594424 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f77257770>: Failed to establish a new connection: [Errno -3] Try again'))

Logger: homeassistant.components.google.calendar
Source: helpers/update_coordinator.py:350
integration: Google Calendar (documentation, issues)
First occurred: 5:35:19 AM (10 occurrences)
Last logged: 7:50:44 AM

Error fetching Restmüll data: Error communicating with API: Error connecting to API: Cannot connect to host www.googleapis.com:443 ssl:default [Try again]
Error fetching Biomüll data: Error communicating with API: Error connecting to API: Cannot connect to host www.googleapis.com:443 ssl:default [Try again]
Error fetching Papiermüll data: Error communicating with API: Error connecting to API: Cannot connect to host www.googleapis.com:443 ssl:default [Try again]
Error fetching Google Privat data: Error communicating with API: Error connecting to API: Cannot connect to host www.googleapis.com:443 ssl:default [Try again]
Error fetching Wertstoffsack data: Error communicating with API: Error connecting to API: Cannot connect to host www.googleapis.com:443 ssl:default [Try again]

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[Skuair]

I do have the same, and like you not sure since when, maybe after core update or host update.

All errors are related to SSL with port 443.
Nothing changed on my setup.

Examples of errors:

Error fetching vigieau api data: Error communicating with API: Cannot connect to host api.vigieau.beta.gouv.fr:443 ssl:default [Try again]

Error requesting waqi data: Cannot connect to host api.waqi.info:443 ssl:default [Try again]

Error fetching co2signal data: Error occurred while communicating to the Electricity Maps API

... Almost all integrations are reporting errors. But they seem to work because I have states on related sensors.

Other information maybe interesting: my CPU load is now 15% when it was 2% idle before last week.

Setup: hassio on RPi 3.

[Phoenix-DH]

I had also just checked my CPU load.
I have also a higher load between 11-19%.
My hardware is a rpi4-64 with 8GB.

Yes it seems that all is working, but openweather on a dashboard is directly visible when connwction has dropped. There are strange values visible until next successful try.

[Phoenix-DH]

Internal mobile app is also affected:

logger:%20homeassistant.components.mobile_app.notify%0ASource%3A%20components/mobile_app/notify.py:203%0Aintegration:%20Mobile%20App%20(documentation,%20issues)%0AFirst%20occurred:%209:10:05%20AM%20(1%20occurrences)%0ALast%20logged:%209:10:05%20AM%0A%0AError%20sending%20notification%20to%20https://mobile-apps.home-assistant.io/api/sendPushNotification:%20ClientConnectorError(ConnectionKey(host='mobile-apps.home-assistant.io',%20port=443,%20is_ssl=True,%20ssl=True,%20proxy=None,%20proxy_auth=None,%20proxy_headers_hash=-8920936574676618749),%20gaierror(-3,%20'Try%20again'))

[szymucha94]

As @Skuair linked above - this is most likely caused by broken supervisor update.
My recommendation is to revert from backup and disable future automatic supervisor updates. Keep in mind this opens supervisor to potential attack vectors unless you keep it isolated from WAN (as you should).

[Phoenix-DH]

Then hopefully next update will solve the issue again.
Thanns for highlighting the root cause, as it seems.

[Rakkzi]

Been having this happen for a couple days now, WAQI, Honeywell Lyric, Emporia Vue, CO2 Signal, Analytics, and Home Assistant Alerts all have 443/SSL errors in the logs and randomly drop out. Devices are failing to authenticate sometimes as well with "Disconnected: Did not receive auth message within 10 seconds".

[AdamGit69]

I am having similar issues on Pi4 8 gig setup, my Notion integration fails reauthentication every 6 or so hours and I notice my weather (BOM integration) and other integrations are loosing connection periodically. Will check my logs when I get a chance later today but I believe it's been happening since the last OS update but I am not 100% sure...

[OGB1952]

@szymucha94 , thanks |I think that's a clue as to what my problem may be.

For me, after updating to 2024.3.1 anything that connects to the internet is now broken, so openweathermap, both the integrations that talk with our cars, the solar generation forecast, basically anything that gets data from the outside world.

The logs are showing no internet connection. Tried reverting to 2024.2 (as the problem started after updating to 2024.3.1) and can't do that as there is no internet connection!

The DNS is OK, double checked and tried changing it to no avail.

It looks like I need to roll back to the previous version that worked, but cannot see anyway to do that without an internet connection. There's no problem with our network or router and none of the setting have changed.

Clearly this won't be fixed by an update, as that cannot happen when the machine can't communicate to the outside world. All local integrations are working fine and I've never had HA set up with an open port to allow external access when away from home.

The machine is an x86 HP thin client, running Home Assistant OS. It's worked fine for well over a year now. I can see there is a very long list of backups, if I type the ha backups command into the terminal, but cannot see how to revert back to one of them.

Sorry for being a bit of a numpty, but I've been at this all day and just cannot find how to get core and supervisor to connect to the internet. If anyone has a suggestion I'd really appreciate it!

If it's any help, this is what the "ha network" command returns:

`➜ ~ ha network info │
docker: │
address: 172.30.32.0/23 │
dns: 172.30.32.3 │
gateway: 172.30.32.1 │
interface: hassio │
host_internet: false │
interfaces: │

• connected: true │
  enabled: true │
  interface: enp1s0 │
  ipv4: │
  address: │
  • 192.168.0.204/24 │
    gateway: 192.168.0.1 │
    method: static │
    nameservers: │
  • 192.168.0.1 │
    ready: true │
    ipv6: │
    address: │
  • fe80::2564:8dcc:848d:bee3/64 │
    gateway: null │
    method: auto │
    nameservers: [] │
    ready: false │
    mac: F4:39:09:46:BD:95 │
    primary: true │
    type: ethernet │
    vlan: null
    wifi: null │
    supervisor_internet: false `

I think the "host_internet: false" and "supervisor_internet: false" messages are the culprit, but have no idea how to fix it.

[Skuair]

From my side not all internet access is broken because HA updates on its own add-ons and my sensor states are correct regarding the integrations that reports SSL errors, so at a moment it works.

I see in the issues list here that several people report integration errors with ip v6, maybe a link.

[OGB1952]

I've tried turning IPv6 off and on again at the router, but it doesn't seem to make any difference. Same goes with changing the DNS settings and getting HAOS to use a specific DNS, rather than that assigned via the router and connection.

I think the clue is that something broke when I updated core to 2024.3.1 as that's when everything that needs to talk with the outside world broke. Right now I have five integrations all reporting they cannot connect and these are the only five integrations that get data via the internet. They are showing as "failed setup, will retry" on the integrations dash, along with a host of failed to connect messages in the logs.

[Phoenix-DH]

I tried also everything related to the router/ip/dns without luck.

you can retry to restart the integrations.
This worked on my end. From time to time, after restart of HA wait some minuted.

All integrations ore online, but offline temporarily.

[OGB1952]

Thanks, @Phoenix-DH , I was getting much the same for the past week, until I updated to 2024.3.1 yesterday. Since then I cannot get anything to connect to the internet at all. Both core_internet and supervisor_internet are constantly reporting false, so something is now permanently broken.

I can't see anything amiss with the router of HAOS settings, nothing has changed and all the non-internet integrations are working fine., and I have a lot of ESP-Home sensors scattered around the place and none of them have missed a beat, so it suggests that it's something specific to do with the way HAOS talks to the internet.

[OGB1952]

An update, I fixed things, but it was more than a bit convoluted.

Not sure how, but during the update to 2024.3.1 somehow the gateway IP address got changed. All a bit odd, as I had been running HA on a static IP for a very long time. The HA IP hadn't changed, but for some unknown reason the gateway IP had, to an IP that doesn't exist on my LAN!

Took a bit of faffing around to edit the gateway IP, essentially I had to delete the static IP, plug the HA machine directly into the router, with everything set to use DHCP, whereupon it duly got itself a new IP and set up the gateway IP to that of the router.

If anyone knows how on earth a gateway IP can get changed to an out of range IP address I'd love to know, just to satisfy my curiosity. FWIW, I wasn't sure this fixed worked at first, as it took around ten minutes for HAOS to sort itself out. Once it had then all the integrations that need to connect to the internet started working as normal. Pleased to say my wife's a happy bunny now, as she can once again see when her car is charging and how much range she's got left . . .

[Phoenix-DH]

Hey @OGB1952 ,

means your issue has gone?
Butvthe solution I did not get.

My HA has a static IP adressed on my routet, but is connected via wire to a unify switch.

On HA itself everything is set to automatic in system - network. If I change it to manual, I get valid values.

Where do you see the wrong gateway?

[Skuair]

All of you have adguard add-on ?
I have it and that was the faulty of my high cpu usage, check hassio-addons/addon-adguard-home#501
Now, my cpu came back to 2-3% like before.

I will check tomorrow about the integrations errors if they are back.

[OGB1952]

@Phoenix-DH

Yes, looks like the problem has gone. I found the oddball gateway IP by using the terminal with the command ha network info

That lists all the network stuff, including the gateway IP address. This should normally be the router IP, as normally that's the gateway to the internet. Took me a while to spot it, as I was concentrating on the two lines in the return from that command relating to both core and supervisor not having access to the internet.

The fix isn't clear, as I tried several different things, but eventually I got it working by disabling the static IP the HAOS machine had been using for a year or so, making sure that DHCP was working, plugging the HAOS machine directly into the router (it had previously been connected to a managed switch).

After rebooting and waiting a fairly long time for the machine to reboot and sort out a connection (no idea why it took so long) I did another ha network info check and found that all was well. Core and supervisor both had internet connectivity and the IP addresses were all as they should be (although the machine IP had changed because it was now dynamic, rather than static.

Sorry this isn't more helpful, I'm afraid I don't understand how on earth the gateway IP changed.

[Phoenix-DH]

I just checked network witrh you command, all is ok.
Due to the fact that I also use ADGuard, my DNS is the IP of HA because I changed it on the router directly.

@Skuair
Thanks for the info, will check.

[Phoenix-DH]

I disabled the setting and since then, no connections errors anymore.
Looking forward if I can enable it back after some new updates.

[Skuair]

I just checked this morning, no more errors as well.
CPU is back to normal.

Note: from my side I did not disable the Adguard setting, I just put my router IP in the field (my router has the hassio IP in its config for the DNS server, don't know if this is the correct setting, but working now).

[OGB1952]

Glad things seem sorted for you both. I wasn't using Adguard, so whatever caused the gateway IP corruption for me was something different, perhaps, but it seems likely that there may have been something during the core update process that inadvertently messed up some network-related settings for all of us having this problem.

I'm back on a static LAN IP again, same one as before, and all still seems to be OK. I might be a bit more circumspect about installing updates in future, perhaps wait a few weeks to see if others have issues before taking the plunge!

[Phoenix-DH]

I just checked this morning, no more errors as well. CPU is back to normal.

Note: from my side I did not disable the Adguard setting, I just put my router IP in the field (my router has the hassio IP in its config for the DNS server, don't know if this is the correct setting, but working now).

Then you also disabled ADGuard, if I get that correct because your clients are not using HA as DNS anymore.

With knowing thast you can play arround in the future.

It is more simple to provide the DNS via your router of the system where ADGUard is running than yonfiguring each client for using it.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.