HA with hassio hangs while updating any node after 1.12.0

[bpmittal]

Operating environment/Installation (Hass.io/Docker/pip/etc.):

hass.io/raspberry pi 3b

ESP (ESP32/ESP8266, Board/Sonoff):
esp32/esp8266/nodemcu/esp01

Affected component:

Description of problem:
i was using ESPHOME successfully but after the latest update 1.12.0 then 1.12.1 , in both the updates whenever i try to update any node then my complete HA hangs with hassio , and restarts automatically , i have tried to update 4-5 nodes but same problem in all ,

one thing i have noticed is if i again try to update the same node then it updates easily ,

please resolve this problem , i cant take any log as my complete system hangs .

Problem-relevant YAML-configuration entries:

PASTE YAML FILE HERE

it is not related to a single yaml file 

Traceback (if applicable):

INFO Reading configuration...
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Compiling app...
INFO Running:  platformio run -d /config/esphome/emergency_light
Processing emergency_light (platform: espressif8266@1.8.0; framework: arduino; board: esp01_1m)
--------------------------------------------------------------------------------
LibraryManager: Installing id=306 @ 1.1.1
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=305
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=1826
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=305 @ 1.1.3
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing v1.12.1
Downloading  [####################################]  100%          
Unpacking  [####################################]  100%
LibraryManager: Installing id=3837 @ 5.13.3
Downloading...
Unpacking  [####################################]  100%
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp01_1m.html
PLATFORM: Espressif 8266 > Espressif Generic ESP8266 ESP-01 1M
HARDWARE: ESP8266 80MHz 80KB RAM (1MB Flash)
Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF MODES: FINDER(chain) COMPATIBILITY(soft)
Collected 29 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ESP Async WebServer> 1.1.1
|   |-- <ESPAsyncTCP> 1.2.0
|   |   |-- <ESP8266WiFi> 1.0
|   |-- <Hash> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ArduinoJson-esphomelib> 5.13.3
|-- <ESP8266WiFi> 1.0
|-- <ESP8266mDNS>
|   |-- <ESP8266WiFi> 1.0
|-- <ESPAsyncTCP> 1.1.3
|   |-- <ESP8266WiFi> 1.0
|-- <Hash> 1.0
|-- <esphome-core> 1.12.1
|   |-- <ArduinoJson-esphomelib> 5.13.3
|   |-- <ESP Async WebServer> 1.1.1
|   |   |-- <ESPAsyncTCP> 1.2.0
|   |   |   |-- <ESP8266WiFi> 1.0
|   |   |-- <Hash> 1.0
|   |   |-- <ESP8266WiFi> 1.0
|   |   |-- <ArduinoJson-esphomelib> 5.13.3
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS>
|   |   |-- <ESP8266WiFi> 1.0
|   |-- <ESPAsyncTCP> 1.2.0
|   |   |-- <ESP8266WiFi> 1.0
Compiling /data/emergency_light/.pioenvs/emergency_light/src/main.cpp.o
Generating LD script /data/emergency_light/.pioenvs/emergency_light/ld/eagle.app.v6.common.ld
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/BearSSLHelpers.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/CertStoreBearSSL.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/ESP8266WiFi.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/ESP8266WiFiAP.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/ESP8266WiFiGeneric.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/ESP8266WiFiMulti.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/ESP8266WiFiSTA.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/ESP8266WiFiScan.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiClient.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiClientSecureAxTLS.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiClientSecureBearSSL.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiServer.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiServerSecureAxTLS.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiServerSecureBearSSL.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib4d9/ESP8266WiFi/WiFiUdp.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib5bb/ESPAsyncTCP_ID305/AsyncPrinter.cpp.o
Archiving /data/emergency_light/.pioenvs/emergency_light/lib4d9/libESP8266WiFi.a
Compiling /data/emergency_light/.pioenvs/emergency_light/lib5bb/ESPAsyncTCP_ID305/ESPAsyncTCP.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib5bb/ESPAsyncTCP_ID305/ESPAsyncTCPbuffer.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib5bb/ESPAsyncTCP_ID305/SyncClient.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib5bb/ESPAsyncTCP_ID305/tcp_axtls.c.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib6f5/Hash/Hash.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/lib6f5/Hash/sha1/sha1.c.o
Archiving /data/emergency_light/.pioenvs/emergency_light/libebd/libArduinoJson-esphomelib_ID3837.a
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/AsyncEventSource.cpp.o
Archiving /data/emergency_light/.pioenvs/emergency_light/lib5bb/libESPAsyncTCP_ID305.a
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/AsyncWebSocket.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/SPIFFSEditor.cpp.o
Archiving /data/emergency_light/.pioenvs/emergency_light/lib6f5/libHash.a
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/WebAuthentication.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/WebHandlers.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/WebRequest.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/WebResponses.cpp.o
Compiling /data/emergency_light/.pioenvs/emergency_light/libefb/ESP Async WebServer_ID306/WebServer.cpp.o

Additional information and things you've tried:

19-03-22 17:44:23 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
19-03-22 18:19:55 WARNING (SyncWorker_16) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_11) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_8) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_7) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_18) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_17) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_1) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_19) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:55 WARNING (SyncWorker_2) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:56 WARNING (SyncWorker_14) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:56 WARNING (SyncWorker_10) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:56 WARNING (SyncWorker_6) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:56 WARNING (SyncWorker_0) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:56 WARNING (SyncWorker_12) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:56 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token
19-03-22 18:19:57 WARNING (SyncWorker_11) [urllib3.connectionpool] Connection pool is full, discarding connection: localhost
19-03-22 18:19:57 INFO (MainThread) [hassio.homeassistant] Updated Home Assistant API token

[brandond]

whenever i try to update any node then my complete HA hangs with hassio , and restarts automatically

Sounds like you're running out of memory on the rPi.

[OttoWinter]

That is very likely not esphome, at least not directly. The message you're seeing is from urllib3, a python HTTP library. esphome uses TCP directly, so the warnings are not (directly) caused by the esphome integration.

Also the connection that is to the loopback interface (not esphome), some other integration must be creating tons of connections to localhost.

[bpmittal]

That is very likely not esphome, at least not directly. The message you're seeing is from urllib3, a python HTTP library. esphome uses TCP directly, so the warnings are not (directly) caused by the esphome integration.

Also the connection that is to the loopback interface (not esphome), some other integration must be creating tons of connections to localhost.

but this problem came only after updating to esphome version 1.12 ,
before that all my nodes were working ok ,
and this error only comes when compiling and node .
this is a serious issue .
my complete ha restarts and all my connected devices restarts .
i cant even make a new node ,
and there is plenty of space left in my hassio server .
and we have not installed and new addon after that ,

if u need any log or data then i can do as required

INFO Reading configuration...
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Compiling app...
INFO Running:  platformio run -d /config/esphome/back_door
Processing back_door (platform: espressif8266@1.8.0; framework: arduino; board: nodemcuv2)
--------------------------------------------------------------------------------
LibraryManager: Installing id=306 @ 1.1.1
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=305
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=1826
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=305 @ 1.1.3
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing v1.12.1
Downloading...
Unpacking  [####################################]  100%
LibraryManager: Installing id=3837 @ 5.13.3
Downloading...
Unpacking  [####################################]  100%
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/nodemcuv2.html
PLATFORM: Espressif 8266 > NodeMCU 1.0 (ESP-12E Module)
HARDWARE: ESP8266 80MHz 80KB RAM (4MB Flash)
Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF MODES: FINDER(chain) COMPATIBILITY(soft)
Collected 29 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ESP Async WebServer> 1.1.1
|   |-- <ESPAsyncTCP> 1.2.0
|   |   |-- <ESP8266WiFi> 1.0
|   |-- <Hash> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ArduinoJson-esphomelib> 5.13.3
|-- <ESP8266WiFi> 1.0
|-- <ESP8266mDNS>
|   |-- <ESP8266WiFi> 1.0
|-- <ESPAsyncTCP> 1.1.3
|   |-- <ESP8266WiFi> 1.0
|-- <Hash> 1.0
|-- <esphome-core> 1.12.1
|   |-- <ArduinoJson-esphomelib> 5.13.3
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS>
|   |   |-- <ESP8266WiFi> 1.0
|   |-- <ESP Async WebServer> 1.1.1
|   |   |-- <ESPAsyncTCP> 1.2.0
|   |   |   |-- <ESP8266WiFi> 1.0
|   |   |-- <Hash> 1.0
|   |   |-- <ESP8266WiFi> 1.0
|   |   |-- <ArduinoJson-esphomelib> 5.13.3
|   |-- <ESPAsyncTCP> 1.2.0
|   |   |-- <ESP8266WiFi> 1.0
Compiling /data/back_door/.pioenvs/back_door/src/main.cpp.o
Generating LD script /data/back_door/.pioenvs/back_door/ld/eagle.app.v6.common.ld
Compiling /data/back_door/.pioenvs/back_door/lib4d9/ESP8266WiFi/BearSSLHelpers.cpp.o
Compiling /data/back_door/.pioenvs/back_door/lib4d9/ESP8266WiFi/CertStoreBearSSL.cpp.o
Compiling /data/back_door/.pioenvs/back_door/lib4d9/ESP8266WiFi/ESP8266WiFi.cpp.o
Compiling /data/back_door/.pioenvs/back_door/lib4d9/ESP8266WiFi/ESP8266WiFiAP.cpp.o
Compiling /data/back_door/.pioenvs/back_door/lib4d9/ESP8266WiFi/ESP8266WiFiGeneric.cpp.o
Compiling /data/back_door/.pioenvs/back_door/lib4d9/ESP8266WiFi/ESP8266WiFiMulti.cpp.o

[OttoWinter]

I know what the warning means: your hassio system has run out of resources (RAM in this case).

And I know why you're seeing it during OTA too: During the OTA the ESP is not active and thus HA tries to reconnect to it, thus opening a new socket.

I also know that it has been like that since the very first esphome version in HA - HA tries to reconnect every minute.

Why are you seeing it now? Maybe a HA update or a newly added component is taking up more open sockets, it could be many things. The OTA producing these warnings is only a side effect of a larger internal issue with your system: something's taking up lots of resources and your system has too little resources.

[bpmittal]

I know what the warning means: your hassio system has run out of resources (RAM in this case).

And I know why you're seeing it during OTA too: During the OTA the ESP is not active and thus HA tries to reconnect to it, thus opening a new socket.

I also know that it has been like that since the very first esphome version in HA - HA tries to reconnect every minute.

Why are you seeing it now? Maybe a HA update or a newly added component is taking up more open sockets, it could be many things. The OTA producing these warnings is only a side effect of a larger internal issue with your system: something's taking up lots of resources and your system has too little resources.

after removing all the addons which i have installed in my raspberry pi 3b+ , not esphome is compiling and is working ok,
but y did that happen ?

i have not installed any addon after the 1.12 update , and i am using esphome in a new raspberry pi 3b+ , with 32gb memory card installed which is 70% free, ram is inbuilt in RPI ,

does this means we will not be able to use RPI for this HA and ESPHOME ?
as it will later cause this error when i will have to install more addons ,

what can i do ?
is there any solution ?

[arretx]

I'm actually experiencing something similar.

When I try to upload, or compile a new node, not 4 lines or so in HA hangs and I have to unplug the Pi3 and reboot. SSH dies, Samba dies, HA dies.

I don't think it's even getting to the communication stage before it crashes.

[brandond]

As mentioned above, it sounds like your Pi doesn't have enough memory to run everything you've put on it and also compile esphome. Compilation takes a lot of CPU and RAM; if you use all the memory on the system it will either hang due to lack of resources, or invoke the OOM killer and start terminating processes.

[OttoWinter]

Agree, RPi does not have enough resources to run ESPHome if the user is also running a lot of addons. That's just the nature of the plugin and due to the lack of resources (probably more about RAM) on the board.

I'm going to close this issue with the note that it's not recommended to run esphome on such low-memory devices. If someone is experiencing issues, they should try disabling all addons that use up lots of memory.

[arretx]

Thank you Otto. My days of running HASS.io on the RPi3 platform have come to an end. I have switched to HASS.io under Ubuntu and quite honestly, it blazes compared to the Pi3. If you're a beginner and you just want some basic functionality, the Pi3 is a great way to go, but if you really want to dig in and run all sorts of additional components, go with a more powerful system.

[r-jordan]

Hi,
I'm sorry for the late comment.
You can also activate a swap file in the RPi3 to make some space in memory. It's enough having a file of 512MB in size as swap. I did it and there's usually 200MB swaped out thus there's more memory for the running processes and I run ESPHome without the OOM kills any process.
Have a good ESPHoming!