Client.get_domain() doesn't return before eating 4+ GB of RAM

[kdknigga]

Describe the bug
Running Client.get_domain() pins a core on my laptop to 100% and starts eating RAM. I had to Ctrl-C to get it to stop.

To Reproduce

import os
import logging

logging.basicConfig(level=logging.DEBUG)

from homeassistant_api import Client as HASSClient

HASS_SERVER = os.getenv("HASS_SERVER")
HASS_TOKEN = os.getenv("HASS_TOKEN")

client = HASSClient(f"{HASS_SERVER}/api", HASS_TOKEN)

client.check_api_running()

lights = client.get_domain("light")     

Expected behavior
I expect some object with all of my lights in it.

Screenshots

$ python                                                                                     
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux                         
Type "help", "copyright", "credits" or "license" for more information.                                                
>>> import os                                                                                                         
>>> import logging                                                                                                     
>>>                                                                                                                   
>>> logging.basicConfig(level=logging.DEBUG)                                                                           
>>>                                                                                                                   
>>> from homeassistant_api import Client as HASSClient                                                                 
>>>                                                                                                                   
>>> HASS_SERVER = os.getenv("HASS_SERVER")                                                                             
>>> HASS_TOKEN = os.getenv("HASS_TOKEN")                                                                              
>>>                                                                                                                    
>>> client = HASSClient(f"{HASS_SERVER}/api", HASS_TOKEN)                                                             
>>>                                                                                                                    
>>> client.check_api_running()                                                                                        
DEBUG:homeassistant_api.rawclient:GET request to http://homeassistant.local:8123/api/                                  
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): homeassistant.local:8123                                
DEBUG:urllib3.connectionpool:http://homeassistant.local:8123 "GET /api/ HTTP/1.1" 200 35                               
DEBUG:homeassistant_api.processing:Using processor <function process_json at 0x7f5d1f028ee0> on <Response [200]>       
True                                                       
>>> lights = client.get_domain("light")
DEBUG:homeassistant_api.rawclient:GET request to http://homeassistant.local:8123/api/services                          
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): homeassistant.local:8123                                
DEBUG:urllib3.connectionpool:http://homeassistant.local:8123 "GET /api/services HTTP/1.1" 200 17322                    
DEBUG:homeassistant_api.processing:Using processor <function process_json at 0x7f5d1f028ee0> on <Response [200]>       
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/venv/lib/python3.10/site-packages/homeassistant_api/rawclient.py", line 213, in get_domain                
    domains = self.get_domains()
  File "/tmp/venv/lib/python3.10/site-packages/homeassistant_api/rawclient.py", line 209, in get_domains               
    return tuple(services)                                 
  File "/tmp/venv/lib/python3.10/site-packages/homeassistant_api/mixins.py", line 18, in process_services_json         
    domain.add_service(service_id, **data)
  File "/tmp/venv/lib/python3.10/site-packages/homeassistant_api/models/domains.py", line 24, in add_service           
    service_id: Service(                                   
  File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__                                               
  File "pydantic/main.py", line 1038, in pydantic.main.validate_model                                                  
  File "pydantic/fields.py", line 857, in pydantic.fields.ModelField.validate                                          
  File "pydantic/fields.py", line 1074, in pydantic.fields.ModelField._validate_singleton                              
  File "pydantic/fields.py", line 1121, in pydantic.fields.ModelField._apply_validators                                
  File "pydantic/class_validators.py", line 313, in pydantic.class_validators._generic_validator_basic.lambda12        
  File "pydantic/main.py", line 679, in pydantic.main.BaseModel.validate                                               
  File "pydantic/main.py", line 605, in pydantic.main.BaseModel._copy_and_set_values                                   
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 172, in deepcopy                                                          
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib64/python3.10/copy.py", line 271, in _reconstruct                                                      
    state = deepcopy(state, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 172, in deepcopy                                                          
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib64/python3.10/copy.py", line 271, in _reconstruct                                                      
    state = deepcopy(state, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 172, in deepcopy                                                          
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib64/python3.10/copy.py", line 271, in _reconstruct                                                      
    state = deepcopy(state, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)                                    
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict                                                    
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 172, in deepcopy                                                          
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib64/python3.10/copy.py", line 271, in _reconstruct                                                      
    state = deepcopy(state, memo)
  File "/usr/lib64/python3.10/copy.py", line 146, in deepcopy                                                          
    y = copier(x, memo)
  File "/usr/lib64/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib64/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/lib64/python3.10/copy.py", line 273, in _reconstruct
    y.__setstate__(state)
KeyboardInterrupt
>>>

Desktop (please complete the following information):

• OS: Fedora
• OS Version: 35
• Python version: 3.10.4
• pip freeze:

aiodns==3.0.0
aiohttp==3.8.1
aiohttp-client-cache==0.6.1
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
Brotli==1.0.9
cattrs==1.10.0
cchardet==2.1.7
certifi==2022.6.15
cffi==1.15.0
charset-normalizer==2.0.12
frozenlist==1.3.0
homeassistant-api==3.0.3.post5
idna==3.3
itsdangerous==2.1.2
multidict==6.0.2
pycares==4.1.2
pycparser==2.21
pydantic==1.9.1
python-forge==18.6.0
requests==2.28.0
requests-cache==0.9.4
simplejson==3.17.6
six==1.16.0
typing_extensions==4.2.0
url-normalize==1.4.3
urllib3==1.26.9
yarl==1.7.2

Home Assistant Versions:

• Home Assistant Core 2022.6.6
• Home Assistant Supervisor 2022.05.3
• Home Assistant OS 8.2

[kdknigga]

I did a little digging. It seems that pydantic==1.9.1 is the problem. Downgrading to 1.9.0 seems to work.

Reference: pydantic/pydantic#4160

[GrandMoff100]

Oh boy, that's a big problem.

But I think I know what the problem is, when services are being assigned to their domain objects they are validating their domain object attribute which in turn is recursively validating themselves again which then validates their domain object again.

TL;DR Trying to deepcopy a self-reference loop.

[GrandMoff100]

Nevermind!

[GrandMoff100]

@kdknigga are there any other concerns you have before I merge my changes?

[kdknigga]

I don't think so.

Thanks for taking a look!

[GrandMoff100]

No problem!