New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gevent.local.local() memory leak #981

Closed
sublee opened this Issue Jun 14, 2017 · 5 comments

Comments

Projects
None yet
3 participants
@sublee
Contributor

sublee commented Jun 14, 2017

  • gevent version: 1.2.2
  • Python version: CPython 2.7.12, PyPy 5.8.0-2.7.13
  • Operating System: Ubuntu 16.04

Description:

There's a memory leak when making gevent.local.local() In a greenlet.

I digged a bit. The clear callback in gevent/local.py#L208 looks like the leak point. Perhaps this issue appears from 1192b1c.

What I've run:

You can reproduce the memory leak by the below simple code:

import gc
import gevent.local
def main():
    while True:
        gevent.local.local()
        gc.collect()
gevent.spawn(main).join()

The growth is much faster on PyPy.

@jamadden

This comment has been minimized.

Member

jamadden commented Jun 14, 2017

It is by design that the local dictionaries aren't cleared until the greenlet exits, so I don't believe that's a leak. If you allocate and never destroy the greenlet they will not be cleared. I believe the prior implementation had the same behaviour (in fact it may have been even less timely).

Most locals are allocated at a static scope (e.g., module or class level) and assigned to a variable from which they are reachable, or occasionally they are allocated as instance attributes, and again, assigned to a variable. They have to be named to be useful, so simply allocating them in a loop is not a good representation of a real scenario.

@sublee

This comment has been minimized.

Contributor

sublee commented Jun 14, 2017

As you said, we can make a local for an instance attribute. At this case, if we have some infinite greenlets that have touched the local at least once, the callbacks from the local will never be released even though the instance is deleted.

I just wanted to show you the memory leak from deleted locals with the shortest code. So I put the allocation without name in the loop. The below example would be more practical:

import gc
import gevent.local

class O(object):
    def __init__(self):
        self.local = gevent.local.local()

def main_loop():
    x = 0
    objs = []
    while True:
        o = O()
        objs.append(o)
        x += 1
        if x % 1000 == 0:
            del objs[:]
            gc.collect()

gevent.spawn(main_loop).join()

jamadden added a commit that referenced this issue Jun 14, 2017

jamadden added a commit that referenced this issue Jun 14, 2017

@sublee

This comment has been minimized.

Contributor

sublee commented Jun 14, 2017

Awesome! Thank you so much!

@hellobiek

This comment has been minimized.

hellobiek commented Aug 31, 2018

Has this problem been solved?

@jamadden

This comment has been minimized.

Member

jamadden commented Aug 31, 2018

As can be seen above, this issue was closed as fixed with the merge of #983

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment