Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
DAL object leaks memory #427
I have an application that runs on CherryPy (v8.1.2) on Ubuntu (14.04.5 LTS) and python 2.7.6. I use standalone pyDAL (v16.09) to connect to my MySQL DB on AWS RDS. I'm seeing a constant memory leak on my servers which causes the process to use up 100% of RAM in a few hours and eventual die a painful death (no SWAP enabled; enabling SWAP pushes back certain death a little).
I have managed to reproduce the leak with a very small loop that repeatedly opens and closes the DAL object. I confirmed that memory continues to grow using htop and will eventually end up using 100% of RAM. I thought it was an MySQL adapter specific problem but I was able to repro it with the default sqlite adapter as well.
I read some really old threads on this issue that suggested calling
More data requested by @mdipierro:
I just ran a few tests and I can confirm that the leak does repro on Mac OS X El Capitan (10.11.6 (15G1004)) with the latest version of pyDAL (16.9) and the HEAD and latest tags for web2py repo (https://github.com/web2py/web2py/releases). The leak did not repro from the last pypi version of web2py (2.1.1). I ran the program I listed in PyCharm on Mac OSX and watched the memory usage using htop installed on the Mac using Homebrew.
Here are the detailed results:
Memory usage at ~4 mins: 193M (https://www.dropbox.com/s/vj5v6hqya4mot2v/Screenshot%202016-11-08%2011.43.28.png?dl=0)
Memory usage at ~5 mins: 16.7M (https://www.dropbox.com/s/yjy8vv7fy7i79b0/Screenshot%202016-11-08%2012.14.52.png?dl=0)
Memory usage at ~4 mins: 187M (https://www.dropbox.com/s/5hx01xve1mpemg5/Screenshot%202016-11-08%2012.32.54.png?dl=0)
@toorsukhmeet Can you please share:
from gluon import DAL sql = DAL() def open_close_dal(): sql._adapter.reconnect() sql.close() if __name__ == '__main__': while True: open_close_dal()
Seems somehow related to pools, but you're not using them. It feels a bit tricky.
The question I have is if the sql instances can be shared between threads this way (after reconnecting). I have a workaround on my server now which creates a thread local instance of
@toorsukhmeet ok found the leak.
@mdipierro this is because every
I've tested this:
def run_leak(n=100): for _ in xrange(0, n): db = DAL('sqlite:memory') db.close()
with some profiling on the memory usage:
and is quite obvious the
>>> from pydal.connection import THREAD_LOCAL >>> dir(THREAD_LOCAL) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_pydal_connection_4390457232_66113', '_pydal_cursors_4390457232_66113', '_pydal_db_instances_', '_pydal_db_instances_zombie_'] >>> run_leak(100000) >>> len(dir(THREAD_LOCAL)) 68690 >>> dir(THREAD_LOCAL)[:30] ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_pydal_connection_4381352592_66113', '_pydal_connection_4381352720_66113', '_pydal_connection_4381352848_66113', '_pydal_connection_4381352912_66113', '_pydal_connection_4388182992_66113', '_pydal_connection_4388822160_66113', '_pydal_connection_4388822288_66113', '_pydal_connection_4388872336_66113', '_pydal_connection_4390457168_66113', '_pydal_connection_4390457232_66113', '_pydal_connection_4390495376_66113', '_pydal_connection_4391236048_66113', '_pydal_connection_4391239056_66113', '_pydal_connection_4391239120_66113', '_pydal_connection_4391239184_66113']
So this is happening on web2py because it creates a DAL instance per-request. On weppy the DAL instance is the same across the processes and is initialized when the server start, this is why I missed the leak.
The proposal is to edit the
def close(self): self._adapter.close() if self._db_uid in THREAD_LOCAL._pydal_db_instances_: db_group = THREAD_LOCAL._pydal_db_instances_[self._db_uid] db_group.remove(self) if not db_group: del THREAD_LOCAL._pydal_db_instances_[self._db_uid] # HERE: some magic to clean connections and cursors too
@mdipierro what do you think?