Memory leak

[mikesalz]

The problem

In the last few days I have noticed a terrible memory leak. RAM usage in increasing at the rate of almost 1 mb per second. If I don’t restart every few hours, HA gets up into the GBs in memory usage and crashes.

I am not sure exactly when this started, but I keep HA pretty updated. I am currently on core-2021.3.4. Running in Docker on a Synology NAS.

I don't see anything weird in the logs.

What is version of Home Assistant Core has the issue?

core-2021.3.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Unknown

Link to integration documentation on our website

No response

Example YAML snippet

# Put your YAML below this line

Anything in the logs that might be useful for us?

# Put your logs below this line

[sj3fk3]

See the same here.. Had more reboots the last couple of days than in the whole of last year (and I was home a lot)

[Sammerdan]

Same here,

I've opened an topic in HA community.

I narrowed it down for now to : service set_location
and something called list
The log I've got is:

`Logger: homeassistant.components.profiler
Source: components/profiler/init.py:221
Integration: Profiler (documentation, issues)
First occurred: 17:19:26 (322 occurrences)
Last logged: 19:59:59

Memory Growth: []
Memory Growth: [('Location', 271488, 1768), ('list', 289129, 1762), ('dict', 56887, 67), ('LocationGroup', 8484, 24), ('Astral', 707, 2), ('AstralGeocoder', 707, 2)]
Memory Growth: [('dict', 56922, 35), ('list', 289147, 18), ('Template', 22, 1)]
Memory Growth: [('Location', 273024, 1536), ('list', 290653, 1506), ('dict', 57013, 91), ('LocationGroup', 8532, 48), ('Astral', 711, 4), ('AstralGeocoder', 711, 4)]
Memory Growth: [('Location', 274176, 1152), ('list', 291796, 1143), ('dict', 57088, 75), ('LocationGroup', 8568, 36), ('Astral', 714, 3), ('AstralGeocoder', 714, 3)]
`

[hruzipet]

I have the same problem. Looks like the problem started with 2021.3.x
I've read previous posts where was mentioned that problem is caused by Brother integration, but Brother integration removed and the problem remains.

[mikesalz]

I have done a lot of testing in the last couple of days. I tried removing integrations that I thought might be the cause, but that did not help. It seems that every time anything happens (a sensor is triggered, a light is turned on, etc) the amount of memory used ticks up and is never released. I tested this with multiple integrations, both native and HACS. Same results. I am now restarting my HA container every 6 hours to prevent it from crashing.

@bdraco You have been incredibly helpful in the past. Do you have any thoughts or suggestions here?

[Kolunchik]

Same here

[bdraco]

The leak is defiantly something to do with astral

./lib/python3.8/site-packages/astral.py:class LocationGroup(object):
./lib/python3.8/site-packages/astral.py:            group = LocationGroup(location._timezone_group)

[bdraco]

Which integrations do you have installed/used?

[mikesalz]

I have listed all of my integrations below.

I did notice something interesting in my testing today. I'm running HA in Docker. I discovered if I set a hard memory limit, the HA container will eventually reach that limit and then hover there. I expected it to hit the limit and crash, forcing the container to restart. That never happened. However, with no limit the container will continue to use more and more memory until it freezes. So for now, I guess my hacky solution is to keep the memory limit in place.

Also, here is my Profiler log output from the other day:

Logger: homeassistant.components.profiler
Source: components/profiler/init.py:221
Integration: Profiler (documentation, issues)
First occurred: 12:48:11 PM (1020 occurrences)
Last logged: 9:17:44 PM

Memory Growth: [('ArloVideo', 16288, 3), ('HassJob', 2437, 2)]
Memory Growth: [('tuple', 54745, 6)]
Memory Growth: [('ArloVideo', 16292, 4)]
Memory Growth: [('method', 6078, 16), ('KeyedRef', 338, 11), ('SelectorKey', 89, 10), ('_SelectorSocketTransport', 78, 10), ('TransportSocket', 80, 10), ('tuple', 54753, 8), ('socket', 99, 7), ('weakref', 16202, 6), ('deque', 667, 6), ('ArloVideo', 16296, 4)]
Memory Growth: [('tuple', 54860, 107), ('cell', 45173, 68), ('function', 73242, 36), ('HassJob', 2453, 16), ('Comparator', 66, 14), ('_truncated_label', 34, 12), ('BindParameter', 56, 11), ('AnnotatedColumn', 28, 10), ('_ColumnEntity', 18, 6), ('_CompileLabel', 18, 6), ('set', 6949, 5), ('partial', 308, 5), ('BinaryExpression', 35, 5), ('_anonymous_label', 14, 5), ('property', 5318, 4), ('itemgetter', 413, 4), ('BooleanClauseList', 14, 4), ('ClauseList', 7, 2), ('Grouping', 3, 2), ('type', 6766, 1), ('QueryEventsDispatch', 4, 1), ('OrderedSet', 11, 1), ('quoted_name', 8, 1), ('SQLiteCompiler', 6, 1), ('prefix_anon_map', 6, 1), ('QueryContext', 3, 1), ('Select', 3, 1), ('Query', 3, 1), ('ResultMetaData', 3, 1), ('Response', 7, 1), ('ArloVideo', 16297, 1)]

Native Integrations

• AccuWeather
• Alexa Media Player
• Apple TV
• August
• Broadlink
• ecobee
• Emulated Roku
• HomeKit Controller
• HomeKit
• Logitech Harmony Hub
• Mobile App
• MyQ
• Obihai
• Omnilogic
• Philips Hue
• Plex Media Server
• Profiler
• Quantum Gateway
• Rainbird
• Ring
• TP-Link Kasa Smart

HACS

• Alexa Media Player
• Fast-Hue polling
• Lutron Caséta Smart Bridge PRO/RA2 Select
• Peloton Sensor
• Arlo Camera Support
• Alarmdotcom

[mikesalz]

I expected it to hit the limit and crash, forcing the container to restart. That never happened.

Eh, I was wrong. It did hold out for quite a while, but it did give up and crash today.

[sj3fk3]

What I find kinda strange: Why are docker only clients effected? I would think if everyone using hass.io / supervisor version would also be effected my guess would be that this would get more priority.

[donkawechico]

I'm seeing this as well and I think it might be due to Apple TV integration, based on what I'm seeing in my logs.

Every day or two, my system will go from around 40% to 100% memory usage over a period of 10 minutes or so. Once it hits 100%, the UI crashes / becomes unavailable, but my ODROID still responds to pings. The only fix is to power cycle the ODROID.

I set up some alerting and was able to watch the issue as it was occurring. Found that my logs were exploding with MRP errors related to Apple TV integration:

File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 340, in _connect
'Connection was re-established to Apple TV "%s"', self.atv.service.name
AttributeError: 'MrpService' object has no attribute 'name'
2021-03-18 13:18:21 ERROR (MainThread) [homeassistant.components.apple_tv] Failed to connect
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 245, in _connect_loop
await self._connect(conf)
File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 340, in _connect
'Connection was re-established to Apple TV "%s"', self.atv.service.name
AttributeError: 'MrpService' object has no attribute 'name'
2021-03-18 13:18:21 ERROR (MainThread) [homeassistant.components.apple_tv] Failed to connect
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 245, in _connect_loop
await self._connect(conf)
File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 340, in _connect
'Connection was re-established to Apple TV "%s"', self.atv.service.name
AttributeError: 'MrpService' object has no attribute 'name'
2021-03-18 13:18:23 ERROR (MainThread) [homeassistant.components.apple_tv] Failed to connect
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 245, in _connect_loop
await self._connect(conf)
File "/usr/src/homeassistant/homeassistant/components/apple_tv/__init__.py", line 340, in _connect
'Connection was re-established to Apple TV "%s"', self.atv.service.name

These log messages repeat a few thousand times as the memory usage climbs.

Of course, these logs could be a red herring. Perhaps some other issue within HA is making Apple TV integration flip out?

@postlund FYI

[postlund]

This is fixed on dev (#47798). Can't tell if it's the root cause here, but can be related as it creates an infinite connect loop in an unpleasant way (see #47711).

[sj3fk3]

It would be nice if we could get an other 2021.3.x release as this is a big issue I think? Even if it's only a week or so to 2021.4

[hruzipet]

I don't think that the problem is related to Apple TV. I'm not using it and I have also problems with a memory leak. Downgrade to 2021.2.3 resolved that problem for now. HA is running in docker.

[Lantamik]

I also see the same thing and am not using any of integrations given above.

[bdraco]

I found an install that is leaking jobs and client sessions.

2021-04-03 16:26:28 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('cell', 57879, 7), ('function', 75151, 3), ('set', 7635, 1), ('HassJob', 4023, 1), ('defaultdict', 3233, 1), ('ClientSession', 3195, 1), ('CookieJar', 3195, 1)]
2021-04-03 16:26:57 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('tuple', 61914, 3)]
2021-04-03 16:27:27 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:27:57 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('cell', 57886, 7), ('function', 75154, 3), ('tuple', 61916, 2), ('set', 7636, 1), ('method', 6273, 1), ('HassJob', 4024, 1), ('defaultdict', 3234, 1), ('ClientSession', 3196, 1), ('CookieJar', 3196, 1), ('CIMultiDict', 3431, 1)]
2021-04-03 16:28:28 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:28:58 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('dict', 78934, 9), ('tuple', 61924, 8), ('cell', 57893, 7), ('method', 6277, 4), ('function', 75157, 3), ('CIMultiDict', 3433, 2), ('set', 7637, 1), ('HassJob', 4025, 1), ('defaultdict', 3235, 1), ('ClientSession', 3197, 1), ('CookieJar', 3197, 1)]
2021-04-03 16:29:28 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:29:57 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:30:28 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('cell', 57905, 12), ('function', 75161, 4), ('set', 7638, 1), ('HassJob', 4026, 1), ('defaultdict', 3236, 1), ('ClientSession', 3198, 1), ('CookieJar', 3198, 1)]
2021-04-03 16:30:58 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:31:28 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:31:58 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('tuple', 61926, 2), ('function', 75163, 2), ('cell', 57907, 2), ('set', 7639, 1), ('HassJob', 4027, 1), ('defaultdict', 3237, 1), ('ClientSession', 3199, 1), ('CookieJar', 3199, 1), ('CIMultiDict', 3434, 1)]
2021-04-03 16:32:28 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('tuple', 61928, 2)]
2021-04-03 16:32:58 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('dict', 78943, 9), ('cell', 57914, 7), ('tuple', 61934, 6), ('function', 75166, 3), ('method', 6280, 3), ('CIMultiDict', 3436, 2), ('set', 7640, 1), ('HassJob', 4028, 1), ('defaultdict', 3238, 1), ('ClientSession', 3200, 1), ('CookieJar', 3200, 1)]
2021-04-03 16:33:28 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:33:58 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('dict', 78947, 4), ('tuple', 61936, 2)]
2021-04-03 16:34:28 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('cell', 57921, 7), ('function', 75169, 3), ('set', 7641, 1), ('HassJob', 4029, 1), ('defaultdict', 3239, 1), ('ClientSession', 3201, 1), ('CookieJar', 3201, 1)]
2021-04-03 16:34:58 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:35:28 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('tuple', 61937, 1)]
2021-04-03 16:35:58 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('cell', 57928, 7), ('tuple', 61941, 4), ('function', 75172, 3), ('set', 7642, 1), ('HassJob', 4030, 1), ('defaultdict', 3240, 1), ('ClientSession', 3202, 1), ('CookieJar', 3202, 1), ('CIMultiDict', 3437, 1)]
2021-04-03 16:36:28 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-04-03 16:36:58 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('tuple', 61948, 7), ('cell', 57935, 7), ('dict', 78952, 5), ('function', 75175, 3), ('method', 6283, 3), ('CIMultiDict', 3439, 2), ('set', 7643, 1), ('HassJob', 4031, 1), ('defaultdict', 3241, 1), ('ClientSession', 3203, 1), ('CookieJar', 3203, 1)]
2021-04-03 16:37:28 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []

[bdraco]

Unfortunately the system ran out of ram before I could dump the objects

[pirogoeth]

I've been having memory leak issues for a bit now as well, and have been resolving by restarting daily. For what it's worth, the integrations I've got in common with @mikesalz are:

Native:

• Mobile App
• MyQ
• Ring
• TP-Link Kasa Smart

HACS:

• Alexa Media Player

I'm going to start collecting and tearing through heap dumps here in a bit.

[bdraco]

Please try 2021.5 beta

https://rc.home-assistant.io/blog/2021/04/28/release-20215/#memory-leaks

[pirogoeth]

Hi, sorry for the lack of update. I switched over to the 2021.5 beta and it seems maybe the leak is slower, but it is still there. I can do a more examination and plot some times, if that would be useful.