-
Notifications
You must be signed in to change notification settings - Fork 704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak in Model #418
Comments
I've got the similar problem for running my parser several times in a system using dynet. for a_eval in data: In my case, a saved model is almost 1.5G so I have to delete model in the memory before running a next job(need to load another model) but it was impossible. I have spent a lot of time to solve it temporally because I have thought the memory problem seems to be my system fault. Thank you share it 👍 |
Did you verify it is indeed a python issue and does not happen in cpp? |
By looking at the cpp code, I think this is because there are no destructors for |
Yes I confirmed this is a c++ problem |
So I took a look at this, and it's basically a result of the DyNet memory allocator not having any concept of garbage collection. When a parameter is allocated the amount of memory the memory allocator is using is incremented, but there's no way to free memory (other than freeing all the memory, which would kill all of the parameters that we have loaded). I thought about this a bit, and I think it might be useful to generalize the DyNet memory management so that we can support different types of allocators:
|
Is this done by calling cleanup() declared in init.h? I have a process that needs to repeatedly create and delete independent dynet models. In this case, would I call initialize() and cleanup() repeatedly? |
When running this code :
With an old version (without dynamic memory pool), I get the following error
There are two big problems here :
Is there a way to limit the memory (ideally an option like
--dynet-mem [initial mem]-[maximum-mem]
) ?The text was updated successfully, but these errors were encountered: