-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Circular references to proxy objects cause files to be locked even after leaving scope #684
Comments
Hi jeffrey, The fact that file are still open when del to all object is strange. |
Yes, this sounds like a good way to do it eventually.
Exactly. A strong reference to the RawIO persists in each proxy object, and the memmaps keep the file open. Each proxy object persists, even after
Good point, it may be easier to solve the file lock issue (not the memory leak problem) with a few targeted weakrefs. However, the first few things I tried didn't work. I tried replacing in each proxy object self._rawio = rawio with self._rawio = weakref.proxy(rawio) This had the effect of allowing the file to unlock with blk = neo.io.get_io(filename).read_block(lazy=True) Here I left out the intermediate step of creating This would probably break a lot of existing code, so that's not a good route to go. I briefly thought maybe we needed weakrefs to the proxy objects themselves in I next tried adding a single strong reference to the RawIO in the The best solution I can think of right now, other than a widespread overhaul of relationships that uses weakrefs in all the right places, is to add a Does that sound like a good idea or a bad one? Would it be better to live with the current behavior that locks the file and use manual garbage collection, until relationships can be overhauled? Or is the "quick fix" of adding |
Thanks for this big investigation. I did not figure out alll theses details. The wiser way is working of relationship with weakref, so the Andrew's plan. |
With code like this,
the file
filename
will remain locked until the Python process ends.I believe the cause is circular references to proxy objects, such as this:
python-neo/neo/io/basefromrawio.py
Lines 264 to 265 in 70f7811
These circular references appear all over the place in Neo and are not unique to proxy objects. I suspect that because of them, most Neo objects in memory are not destroyed even after commands like
del io; del blk
. Not only is this a source of a memory leak, but in the case of lazy loading using proxy objects, the file becomes locked (e.g., it cannot be deleted in Windows at least).Manually performing garbage collection is a workaround:
However, the best solution I think would be to use weak references.
The text was updated successfully, but these errors were encountered: