Memory leak occurs when using multiprocessing with large data objects

[abhiramch018]

Bug report

Bug description:

When using the multiprocessing module in Python to handle large data objects, I noticed that the memory usage keeps increasing even after all processes are finished. The memory is not released properly, which suggests a possible memory leak when transferring big objects between processes.

This issue mainly happens when sending large lists or dictionaries through multiprocessing.Queue or Pipe. The child processes seem to hold references even after termination, and garbage collection does not free the memory completely. This causes performance degradation and can lead to crashes in long-running scripts.

CPython versions tested on:

3.12

Operating systems tested on:

Windows

[efimov-mikhail]

Could you please provide an example?
It's always better to have minimal reproducible example.
It'll be difficult to improve something without it.

[picnixz]

Please provide an example of such regression. Without more information we cannot investigate. Please show us numbers to indicate that the child process is indeed "dead" and doesn't live anymore. If objects are sent through Queue or Pipe, there is the following warning:

This means that whenever you use a queue you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined. Otherwise you cannot be sure that processes which have put items on the queue will terminate. Remember also that non-daemonic processes will be joined automatically.

Also, please indicate which start method is being used. And how processes are created, used and joined. Without concrete reproducers, I will close this one because there will nothing we can do on our side without more information and context.