Encryption makes some HA entities "unavailable" after reboot

[truglodite]

The problem

One of my particular esphome devices was having consistent problems with entities showing up as "unavailable" in HA (hassos on a pi4-4gb), despite the device appearing to work fine in the esphome webui. I tried hacking storage files and replacing the device/entities, but the issue kept returning.

In the process of troubleshooting, I tried disabling encryption on the device in question. This seems to have fixed the problem altogether, as the device has gone now a week without issues (after many HA reboots that would have triggered it before). So I am assuming there is a bug/incompatibility with encryption and some of the other features I have implemented on the device. FWIW, I have many other esphome devices that are using encryption without an issue, but the device that is failing does have other features (mainly sensors) enabled that none of the others have.

I'm not familiar enough with esphome and the related coding languages to be able to help fix this, other than providing info/logs etc. Please let me know if there's something else I can provide to help here.

Which version of ESPHome has the issue?

2021.12.3

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2021.12.7

What platform are you using?

ESP8266

Board

wemos d1 mini

Component causing the issue

encryption

Example YAML snippet

substitutions:
  name: "garage"
  friendly_name: "Garage"

# Custom esp12f device
esphome:
  name: ${name}
  platform: ESP8266
  board: d1_mini
  on_boot:
    then:
      - switch.turn_off: beeper
      - switch.turn_off: relay

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "${friendly_name} Fallback Hotspot"
    password: "MoreSecrets"

captive_portal:

# Enable logging to get dallas address
logger:
  
# Enable HA API
api:
  password: !secret garage_password
  encryption:
    key: !secret encryption_key
# Enable ota
ota:
  password: !secret ota_password
# description         GPIO  D1 mini
#-------------------------------------------------
# relay               15    D8     // relay input pin (must stay low on power up)
# motion              13    D7     // motion output pin
# s-Tx                12    D6     // software serial
# s-Rx                14    D5     // software serial
# built in LED        2     D4     // status
# nc                  0     D3     // na
# scl                 4     D2     // i2c clock pin
# sda                 5     D1     // i2c data pin
# beeper              16    D0     // Beeper input pin
  
i2c:  
  sda: GPIO5
  scl: GPIO4
  frequency: 100kHz
  
uart:
  tx_pin: GPIO12
  rx_pin: GPIO14
  baud_rate: 9600

dfplayer:

binary_sensor:
  - platform: status
    name: "${friendly_name} Status"
    
  - platform: gpio
    pin:
      number: GPIO13
      mode: INPUT
    name: "${friendly_name} Motion"
    device_class: motion
    

text_sensor:
  - platform: homeassistant
    name: "ESPhome Voice"
    internal: true
    entity_id: input_text.esphome_voice
    on_value:
      then:
        - dfplayer.set_volume: 25
        - delay: 150ms
        - dfplayer.play_folder:
            folder: 1
            file: !lambda return atoi(x.c_str());
            
sensor:
  - platform: wifi_signal
    name: "${friendly_name} WiFi Signal"
    update_interval: 60s
        
  - platform: bh1750
    name: "${friendly_name} Luminance"
    address: 0x23
    update_interval: 10s
    
  - platform: bme280
    temperature:
      name: "${friendly_name} Temperature"
      oversampling: 16x
    pressure:
      name: "${friendly_name} Pressure"
      filters:
      - lambda: return (x + 3.0);
    humidity:
      name: "${friendly_name} Humidity"
    address: 0x76
    update_interval: 60s
    
status_led:
  pin:
    number: GPIO2
    
switch:
  - platform: gpio
    name: "${friendly_name} Opener Relay"
    pin: GPIO15
    id: relay
    icon: mdi:garage
    on_turn_on:
    - delay: 500ms
    - switch.turn_off: relay
    
  - platform: gpio
    name: "${friendly_name} Beeper"
    pin: GPIO16
    id: beeper
    internal: true
    icon: mdi:volume-low
    
  - platform: template
    name: "${friendly_name} Beeps 2"
    turn_on_action:
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    
  - platform: template
    name: "${friendly_name} Beeps 3"
    turn_on_action:
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    
  - platform: template
    name: "${friendly_name} Beeps 5"
    turn_on_action:
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper
    - delay: 50ms
    - switch.turn_on: beeper
    - delay: 50ms
    - switch.turn_off: beeper

### Anything in the logs that might be useful for us?

_No response_

### Additional information

_No response_

[tht]

I do have the same issue. Most likely the same issue as home-assistant/core#58143.

[raidnet-ms]

I have the same problem.
If I add a new esphome device with encryption key, the device is disconnected from the API after a while so i lose any entity data.
The device will be connected to the wifi and available for OTA. Unfortunately to see entities back again, I must delete the device and add it again. Few minutes later it will be disconnected again. No matter if I restart manually, it will not login back to the API.

Removing the encryption key from the API resolved the issue.

I'm not sure but I think the problem is related with 2021.12.3 because with 2021.12.0 I had no issues.

[rumunn95]

My encryption API is blank and have same issue, have two esp but one after reboot HA makes entity not available, when reboot couple times HA entity show up... log show everything ok but entity not available.
`logger:

Enable Home Assistant API

api:

ota:
password:`

[raidnet-ms]

My encryption API is blank and have same issue, have two esp but one after reboot HA makes entity not available, when reboot couple times HA entity show up... log show everything ok but entity not available. `logger:

Enable Home Assistant API

api:

ota: password:`

are you able to make ota updates?

[rumunn95]

esp is online and visible on HA espHome integration. I able to read logs without problems, yes ota updates work perfect but when i reboot HA all entities shows not available, after i reboot HA several times or disconect power from raspberry pi 3 server entities show up when I am lucky... only problem with one device. I make full backup HA and instal fresh system on SD card restore backup and nothing change ;(

[raidnet-ms]

seems the same problem but without encryption I temporary solved it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[At-M]

Sadly this issue persists, one of my sensors just stopped working and now its entities won't be recognized.

It shows up in the WebGUI as online, i can see the log output , do OTA Updates, and it actually tries to send the messages to Home Assistant, but it does not show data when i have a look at the entities.

this is part of my .yaml, theres no encryption setup and while booting it says

Using noise encryption: NO

# Enable Home Assistant API
api:
  password: !secret xxxapipw

ota:
  password: !secret xxxotapw
  

Home Assistant 2022.9.4
Supervisor 2022.10.2
Operating System 8.4
ESPHome 2022.3.1

[ssieb]

tries to send the messages to Home Assistant

That doesn't actually make sense. The connection is from the HA side, so can you verify that HA is able to connect to the device?

[At-M]

That doesn't actually make sense. The connection is from the HA side

ah okay, i thought it always publishes the message when it reads the sensors, thx for that clarification

sometimes theres something like "HA connected" and after a short time "connection dropped" (can't remember the exact phrase right now), so i guess that's it

can you verify that HA is able to connect to the device?

i can OTA new firmware on it,can see the log in realtime via the HA Addon, and also it shows up as "online" in the ESPHome dashboard

[ssieb]

And none of those last things has anything to do with HA.
You need to check the ha logs for why it's disconnecting.

[leodrivera]

I'm facing several problems with the api encryption (like one said in home-assistant/core#80813). Even updating the api-key or reloading the entity the problem persists. I've been keeping encryption turned off so I can use the devices normally.

[bdraco]

There were a few bugs in the aioesphomeapi code that failed to correctly report encryption configuration and incorrect keys that were fixed in the last few months. If you haven't had a chance to try it again and want to do encryption, you should get a much better experience now.

[raidnet-ms]

Hi Bradco, thanks for your answer. We will try and let you know!