ESPHome devices Entities Unavailable after Updating to 2021.10

[apo-mak]

The problem

All the Entities of any ESPHome Device has stop updating and are now Unavailable.
After updating ESPHome to 2021.10 HA has start to log an error in the logs and dose not update any data (related to ESPHome devices)

What is version of Home Assistant Core has the issue?

core-2021.10.6

What was the last working version of Home Assistant Core?

core-2021.10.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

ESPHome

Link to integration documentation on our website

https://www.home-assistant.io/integrations/esphome/

Example YAML snippet

esphome:
  name: NODENAME
  platform: ESP32
  board: nodemcu-32s

wifi:
  ssid: !secret NODENAME _SSID
  password: !secret PLACEHOLDER_PASS_WIFI

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "NODENAME Fallback Hotspot"
    password: "RANDOM_PASS"

captive_portal:

# Enable logging
logger:

# Enable Home Assistant API
api:
  password: !secret PLACEHOLDER_PASS
  encryption:
    key: !secret PLACEHOLDER_KEY

ota:
  password: !secret PLACEHOLDER_PASS_OTA


#BLE Sensor discovery 
esp32_ble_tracker:
  scan_parameters:
    active: false

Anything in the logs that might be useful for us?

Logger: homeassistant
Source: components/esphome/entry_data.py:146
First occurred: 03:33:01 (1 occurrences)
Last logged: 03:33:01

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/esphome/__init__.py", line 294, in complete_setup
    infos, services = await entry_data.async_load_from_store()
  File "/usr/src/homeassistant/homeassistant/components/esphome/entry_data.py", line 146, in async_load_from_store
    self.device_info = DeviceInfo.from_dict(restored.pop("device_info"))
KeyError: 'device_info'

Additional information

In one of the ESPHome devices i also enable the encryption option and HA asked me to reconfigured twice.

[probot-home-assistant]

esphome documentation
esphome source
_{^{(message by IssueLinks)}}

[probot-home-assistant]

Hey there @OttoWinter, @jesserockz, mind taking a look at this issue as it has been labeled with an integration (esphome) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[apo-mak]

I remove all the esp devices and restarted the host machine of HA and

After re Building the image

selecting " clean build" restart the add-on "install" on esp device.

There are no more errors and i manage to re configure the esp devices.

not sure what cause this in the first place.

[apo-mak]

The problem appears again after each update / flash of the nodes .

After the the nodes update, all ESPHome entity's stay unavailable.
I fond that by

Removing the node that has encryption enabled from the HA integrations then restart HA from the Configuration /Server Controls and re add the node from the discovery notification there is no error on the logs and all the ESPHome entity's 'are providing there data correctly .

It looks to be a problem with the API reconnection if the node has encryption enabled.

[tomlut]

You need to open this issue again if it is not solved.

I have seen similar issues after enabling encryption.

If I do an update all in the ESPHome addon, all the nodes update and communicate with the addon without issue. However some random nodes fail to reconnect to the Home Assistant integration and require reloading of the integration for that node. Then they function as expected.

There is no common factor that I can see. Different devices every time, different board types, no one particular component used.

[matphillips]

I'm seeing the same behaviour. Making a change to the code on a node and refreshing it causes it to show as unavailable in HomeAssistant. Fix is to remove the node and re-add it. It's a big tedious for development!

[tomlut]

You shouldn't have to remove them. Just reload the integration.

[matphillips]

Oh, I hadn't tried that! That's a lot less effort than pasting in the encryption key and API key again and again!
I'll try that next time.
Great tip, but wasn't required previously, so hopefully it can be resolved in a future release.

[timdaman]

I saw this "unavailable" behavior when I updated my nodes to switch from the use of passwords to using encryption (they are functionally redundant). I found my devices stopped working even though ESPHome saw them as fine. I noticed that in my ".storage/core.config_entries" file the password was still there even though I didn't have it configured anymore. When I replaced the password with a blank string on my devices that are using encryption and restarted HA the problem disappeared.

That being said, I am seeing serious issues when I use encryption, devices rebooting frequently and taking a long time to come back online, so I am moving back to passwords.

[JasonSwindle]

To chime in on this; I just moved all of my ESPHOME devices off of encryption, as I having to restart HA at random to make it work. Sadly, there are no errors in the logs to help debug this.

[RaveGun]

I also found out that some of my nodes are no longer available. These are ESPHome nodes that were working reliable since day one.
The only thing I found in the core's logs is something like this:

2021-11-06 10:21:27 WARNING (MainThread) [homeassistant.components.esphome] Error getting initial data for 10.6.66.112: Timeout waiting for response for <class 'api_pb2.DeviceInfoRequest'>
2021-11-06 10:21:43 WARNING (MainThread) [aioesphomeapi.reconnect_logic] Can't connect to ESPHome API for 10.6.66.112 @ 10.6.66.112: Error connecting to ('10.6.66.112', 6053): [Errno 113] Connect call failed ('10.6.66.112', 6053)
2021-11-06 10:23:00 WARNING (MainThread) [aioesphomeapi.reconnect_logic] Can't connect to ESPHome API for 10.6.66.112 @ 10.6.66.112: Timeout waiting for response for <class 'api_pb2.ConnectRequest'>
2021-11-06 10:23:44 WARNING (MainThread) [homeassistant.components.esphome] Error getting initial data for 10.6.66.112: Timeout waiting for response for <class 'api_pb2.DeviceInfoRequest'>
2021-11-06 10:27:40 WARNING (MainThread) [aioesphomeapi.reconnect_logic] Can't connect to ESPHome API for 10.6.66.112 @ 10.6.66.112: Timeout while connecting to ('10.6.66.112', 6053)
2021-11-06 10:31:43 WARNING (MainThread) [homeassistant.components.esphome] Error getting initial data for 10.6.66.112: Connection not done for stroz @ 10.6.66.112!

This is just a grep for a specific IP although there are several nodes affected an the logs are interlaced.

[smiba]

I have an issue that is similar or the same to this, every time I restart home-assistant the second ESPHome device I've added (I have two in total) becomes unavailable and requires me to remove and re-add it. After re-adding it it works again straight away and picks up where it left off (including historic data)

Error in the log is the same:

2021-11-16 22:30:36 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/esphome/__init__.py", line 312, in complete_setup
infos, services = await entry_data.async_load_from_store()
File "/usr/src/homeassistant/homeassistant/components/esphome/entry_data.py", line 146, in async_load_from_store
self.device_info = DeviceInfo.from_dict(restored.pop("device_info"))
KeyError: 'device_info'

My other ESPHome device does not suffer from this issue, I assume because this is the first device I've added. They're otherwise identical smart power plug (same firmware too)

[CyanoFresh]

have this issue with some boards, too. Need to reload integration, which is annoying

[hawkeye217]

I updated 3 different HA instances to 2021.12.5 recently, and on two of them I've had a couple of ESPHome nodes go "unavailable" in HA even though they remain pingable and accessible via their web interface. The only thing that corrected the issue was to reload the integration in HA. I've never had to do this before upgrading.

[nagyrobi]

My observation related to this, using encryption:

• normally I have stable connection with the nodes
• when updating the nodes, at 80% of the cases the node becomes unavailable after OTA finishes and node reboots.
• in such cases, the nodes become unavailable somewhere during the OTA procedure (via network)
• for the rest, 20% of the cases the nodes re-appear after OTA+Reboot if they didn't go unavailable during the OTA

This makes me think that the issue is caused by the fact that when HA periodically queries the node via api, if the query happens during an OTA transfer, the entities go unavailable immediately and never come back.

If the OTA transfer is fast enough so that it can slip in between 2 queries of the API, no problem occurs.

I think there must be something related to API behavior during an OTA transfer.

[otzen]

I had the same problem as described here.
Mine came after a power outage on the house.
I knew the connection to the ESP was fine as i could connect and see the logs from HA.
After a lot of investigation I notiched my esp's has gotten new IP's after the outage (router down too), but it was only the one out of 4 that had problems.
I digged my way into config/.storage/core.config_entries
Here i found the entry for my esp's here is the faulty one:

            {
                "entry_id": "7a1fbc9dd804182d5e3a429e68375e4a",
                "version": 1,
                "domain": "esphome",
                "title": "Loft syd hub",
                "data": {
                    "host": "192.168.1.100",
                    "port": 6053,
                    "password": "",
                    "noise_psk": ""
                },
                "options": {},
                "pref_disable_new_entities": false,
                "pref_disable_polling": false,
                "source": "zeroconf",
                "unique_id": "gaestevaerelse",
                "disabled_by": null
            },

The old ip was still listed there, for the other ESP's the IP was updated automatically.
I changed the ip, restarted HA, and boom the entities was online agian 8-D

I did however notice a hint of a pattern
The IP that failed to auto update was going from 192.168.1.100 to 192.168.1.10
the IPs that suceeded
192.168.1.101 -> 192.168.1.11
192.168.1.102 -> 192.168.1.9
192.168.1.99 -> 192.168.1.13

I was just thinking if you doo a string comparison of "192.168.1.100" and "192.168.1.10", but only consider the lentgh of the latter one. It will claim a match... Just thinking.

[apo-mak]

looks like it is fixed in Home Assistant 2022.3.7 ESPHome device did not require manual reload and all the sensors are updating normally.

[RaveGun]

I updated 2x Gosund SP112 with the Firmware: 2022.3.1 (Mar 24 2022, 18:28:12) just to find out that one was no longer available one day latter. A reload solved the problem. Where is the 2022.3.7 version available? Thanks.

[apo-mak]

I updated 2x Gosund SP112 with the Firmware: 2022.3.1 (Mar 24 2022, 18:28:12) just to find out that one was no longer available one day latter. A reload solved the problem. Where is the 2022.3.7 version available? Thanks.

2022.3.7 was the Home Assistant version not the ESPHome.

Currently i am on HA 2022.3.8 with a ESPHome 2022.3.2 and the problem has not re appeared.
Hope this helps.