Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mypyc causing memory leaks in mypy #555

Closed
msullivan opened this issue Apr 10, 2019 · 4 comments

Comments

Projects
None yet
1 participant
@msullivan
Copy link
Collaborator

commented Apr 10, 2019

I augmented mypy's __main__.py to call main four times and do a gc and a memprofile after each one:

if __name__ == '__main__':
    for i in range(4):
        main(None)
        reset_global_state()
        gc.collect(2)
        print_memory_profile()

Compiled with mypyc, the max RSS and size of reachable objects increases every run, with a large (and growing each time) number of mypy type and ast nodes in the list of reachable objects.

When interpreted, it stays stable, and there aren't mypy type and ast nodes in the list.

@msullivan

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 10, 2019

We have a major leak where we don't ever decref __dict__ on mypyc-compiled objects that have one.

I have a fix for this, but it looks like there is at least one more source of leak.

msullivan added a commit that referenced this issue Apr 10, 2019

Don't incref borrowed values unless they are live
Otherwise we increment borrowed values that will never be decremented

Work towards #555.

msullivan added a commit that referenced this issue Apr 11, 2019

@msullivan

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 11, 2019

I tracked the leaks down to three separate bugs (mypy's data structures are so cyclic that any one leak of a type or ast node will cause a lot of leakage). PRs to fix them are #557, #558, #559.

msullivan added a commit that referenced this issue Apr 11, 2019

Don't incref borrowed values unless they are live (#558)
Otherwise we increment borrowed values that will never be decremented

Work towards #555.

msullivan added a commit that referenced this issue Apr 11, 2019

@msullivan

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 11, 2019

Fixes all merged!

@msullivan msullivan closed this Apr 11, 2019

@msullivan msullivan reopened this Apr 15, 2019

msullivan added a commit that referenced this issue Apr 15, 2019

Don't leak hash values in tp_hash wrappers
I missed this one last week because the leaked objects didn't show up
in gc.get_objects or its graph, and my testcase was small enough that
I didn't notice RSS creeping up. It shows up super clearly in a debug
build using sys.getobjects().

Fixes #555. (Again.)

msullivan added a commit that referenced this issue Apr 15, 2019

Don't leak hash values in tp_hash wrappers (#562)
I missed this one last week because the leaked objects didn't show up
in gc.get_objects or its graph, and my testcase was small enough that
I didn't notice RSS creeping up. It shows up super clearly in a debug
build using sys.getobjects().

Fixes #555. (Again.)
@msullivan

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 16, 2019

There was another leak, fixed in #562

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.