Memory leak in multiprocessing.managers.Server #103136
Description
Bug report
multiprocessing.managers.Server doesn't sufficiently clean up id_to_local_proxy_obj when objects are deleted.
Minimal Example
from multiprocessing.managers import SyncManager
# Setup syncmanager
with SyncManager() as sm:
# Create shared list, object we hold is a proxy to the list
shared_list = sm.list()
# Create shared lock, object we hold is a proxy to the lock
shared_lock = sm.Lock()
# Place the lock (proxy) in the list. The SyncManager server is smart
# enough to notice that this is a proxy to an object that it owns, and
# keeps track of this in id_to_local_proxy_obj
shared_list.append(shared_lock)
print("created objects")
# We no longer need a local copy of the lock (in practice, maybe this has
# gone out of scope etc.)
del shared_lock
print("deleted local lock, is only stored in the shared list")
# Remove the lock from the list. At this point there *should* be no more
# references to the lock or its proxies. However, as the
# id_to_local_proxy_obj is not cleaned up, the object is still there
del shared_list[0]
print("deleted lock from list")
print("server now cleans up and deletes the lock object from id_to_obj")
input()It is difficult for me to find a programmatic way to show the leak, as the Server prevents IPC access to its members. By attaching a debugger and setting a breakpoint on Server.shutdown (eg. multiprocessing.managers:355), you can see that as we exit the contextmanager (and the server is shutdown) the only object it is managing (in self.id_to_obj) is the list. This list is empty, however there is still a Lock object present in self.id_to_local_proxy_obj. Clearly if managed objects are created/added/deleted in significant numbers, then the usage of this secondary dict will keep increasing.
Your environment
- CPython versions tested on: 3.8
- Operating system and architecture:
- Ubuntu Focal 20.04
- Ubuntu Focal 20.04 on WSL (Windows 10)
As the relevant objects have not changed in more recent versions, there is no reason to think that this would be patched in up-to-date CPython. (This is not related to #74442)
I am happy to contribute and write a fix for this, although I may need suggestions for an approach for appropriate test.