New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Identifying a memory leak #538

Closed
raybotha opened this Issue Apr 29, 2018 · 7 comments

Comments

Projects
None yet
3 participants
@raybotha
Copy link
Contributor

raybotha commented Apr 29, 2018

I'm using the nameko backdoor to try and find the cause of a memory leak, although my service is quite simple and only uses the nameko-sqlalchemy dependency, and rpc.

However the memory use balloons up to around 6GB after handling a few thousand requests, but running pympler does not yield any results:

>>> from pympler import muppy
>>> allob = muppy.get_objects()
>>> from pympler import summary
>>> sum = summary.summarize(allob)
>>> summary.print_(sum)
                                          types |   # objects |   total size
=============================================== | =========== | ============
                                    <class 'str |       43963 |      5.76 MB
                                   <class 'dict |        7918 |      3.52 MB
                                   <class 'code |       15101 |      2.08 MB
                                   <class 'type |        2055 |      2.07 MB
                                    <class 'set |        1299 |    502.16 KB
                                  <class 'tuple |        6963 |    472.64 KB
                                   <class 'list |        3412 |    368.48 KB
                                <class 'weakref |        3720 |    290.62 KB
                            <class 'array.array |           2 |    260.79 KB
  <class 'sqlalchemy.sql.visitors.VisitableType |         248 |    256.65 KB
                            function (__init__) |        1145 |    152.07 KB
                            <class 'abc.ABCMeta |         136 |    135.81 KB
                      <class 'getset_descriptor |        1856 |    130.50 KB
                                    <class 'int |        4373 |    122.17 KB
                     <class 'wrapper_descriptor |        1466 |    114.53 KB

Is there a part of the nameko service memory that pympler can't access?

@raybotha

This comment has been minimized.

Copy link
Contributor

raybotha commented Apr 29, 2018

Versions:

nameko==2.8.4
nameko-sqlalchemy==1.1.0
@mattbennett

This comment has been minimized.

Copy link
Contributor

mattbennett commented May 1, 2018

I have had good success diagnosing leaks with pympler via the backdoor. As far as I'm aware it can see everything.

The problem is probably in nameko-sqlalchemy. Which of the two DependencyProviders are you using? The newer one is more efficient if you're handing a lot of concurrent requests.

@mattbennett

This comment has been minimized.

Copy link
Contributor

mattbennett commented May 1, 2018

In the two cases I've diagnosed, the causes were:

  1. A circular reference causing data in the worker context not to be garbage collected
  2. Lots of backed up requests accepted by the @http entrypoint but blocked on the worker pool (not strictly a leak)

In both cases I could see the objects in pympler though.

@mattbennett mattbennett added the question label May 3, 2018

@bludau-peter

This comment has been minimized.

Copy link

bludau-peter commented Jul 25, 2018

Hi all,
I have a similar issue where the RAM usage is about 2GB after 1 long running but finished http request (no other requests made to the service since start) and same results as @raybotha using pympler. My dependencies are 'pymongo', 'nameko==2.9.0', 'pandas', 'numpy'. Does anyone have any idea why this may happen?

@raybotha

This comment has been minimized.

Copy link
Contributor

raybotha commented Jul 25, 2018

I have this in services that don't use mongo, pandas or numpy but a lot of my services consistently build up memory use until they get restarted. Rough figures, at around 1 request per second taking ~50ms it can get up to 3GB after two or three days.

I'm getting this in some services that don't use sqlalchemy, here's a pip freeze for one such service that's acting as an API gateway and not doing much of anything except calling other services:

amqp==1.4.9
anyjson==0.3.3
attrs==17.4.0
certifi==2018.1.18
chardet==3.0.4
ddtrace==0.11.1
enum-compat==0.0.2
eventlet==0.21.0
greenlet==0.4.13
idna==2.6
kombu==3.0.37
mock==2.0.0
msgpack-python==0.5.6
nameko==2.8.5
path.py==11.0.1
pbr==4.0.2
phonenumbers==8.9.3
pycountry==18.2.23
PyYAML==3.12
requests==2.18.4
simplejson==3.13.2
six==1.11.0
toastedmarshmallow==0.2.6
urllib3==1.22
Werkzeug==0.14.1
wrapt==1.10.11
@mattbennett

This comment has been minimized.

Copy link
Contributor

mattbennett commented Jul 28, 2018

@bludau-peter my first investigation would be to find out in what situations a process will consume memory and the objects not be visible in pympler. At a guess it's something to do with c-extensions, in your case either in pandas or numpy. The fact that memory grows after only one worker execution suggests it has nothing to do with Nameko or the request-response cycle.

@mattbennett

This comment has been minimized.

Copy link
Contributor

mattbennett commented Aug 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment