multiprocessing.Pool and ThreadPool leak resources after being deleted

[tzickel]

BPO	34172
Nosy	@pitrou, @vstinner, @benjaminp, @ned-deily, @zware, @mattip, @MojoVampire, @applio, @tzickel, @pablogsal, @Windsooon
PRs	bpo-34172: multiprocessing.Pool leaks resources after being deleted #8450 [3.7] bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) #9676 [3.6] bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) #9677 [2.7] bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) #9686 [3.7] Revert "bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) (GH-9676)" #10968 [3.6] Revert "bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450) (GH-9677)" #10969 [2.7] Revert "[2.7] bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-9686)" #10970 Revert "bpo-34172: multiprocessing.Pool leaks resources after being deleted (GH-8450)" #10971

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-07-20.16:45:29.720>
labels = ['3.8', 'type-bug', 'library', 'docs']
title = 'multiprocessing.Pool and ThreadPool leak resources after being deleted'
updated_at = <Date 2019-10-23.18:45:00.937>
user = 'https://github.com/tzickel'

bugs.python.org fields:

activity = <Date 2019-10-23.18:45:00.937>
actor = 'josh.r'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)']
creation = <Date 2018-07-20.16:45:29.720>
creator = 'tzickel'
dependencies = []
files = []
hgrepos = []
issue_num = 34172
keywords = ['patch']
message_count = 52.0
messages = ['322028', '322076', '322090', '322129', '322147', '322150', '322172', '322190', '322194', '322230', '322231', '323103', '323104', '326904', '326909', '326913', '326914', '326915', '326948', '326964', '330864', '330865', '330866', '330868', '330869', '330891', '330954', '330955', '330960', '330962', '330988', '330996', '330997', '330999', '331009', '331026', '331087', '331190', '331198', '331199', '331200', '331210', '331216', '331218', '331221', '331629', '331703', '332294', '335357', '355229', '355247', '355248']
nosy_count = 13.0
nosy_names = ['pitrou', 'vstinner', 'benjamin.peterson', 'ned.deily', 'docs@python', 'zach.ware', 'mattip', 'josh.r', 'davin', 'tzickel', 'pablogsal', 'Windson Yang', 'Vy Nguyen']
pr_nums = ['8450', '9676', '9677', '9686', '10968', '10969', '10970', '10971']
priority = None
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34172'
versions = ['Python 3.8']

[tzickel]

In multiprocessing.Pool documentation it's written "When the pool object is garbage collected terminate() will be called immediately.":

https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing.pool.Pool.terminate

A. This does not happen, creating a Pool, deleting it and collecting the garbage, does not call terminate.

B. The documentation for Pool itself does not specify it has a context manager (but the examples show it).

C. This bug is both in Python 3 & 2.

[Windsooon]

A. This does not happen, creating a Pool, deleting it and collecting the garbage, does not call terminate.

Would you give me an example how you delete the Pool and collecting the garbage? If you use context manager, It will call terminate() function.

B. The documentation for Pool itself does not specify it has a context manager (but the examples show it).

You can find this info on the same page:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

[tzickel]

>>> from multiprocessing import Pool
>>> import gc
>>> a = Pool(10)
>>> del a
>>> gc.collect()
0
>>>

After this, there are still left behind Process (Pool) or Dummy (ThreadPool) and big _cache data (If you did something with it) which lingers till the process dies.

You are correct on the other issue (I'm using and reading the Python 2 documentation which does not have that...).

[Windsooon]

A patch would just add

    def __del__(self):
        self.terminate()

in the Pool object.

[tzickel]

But alas that does not work...

[Windsooon]

Add a __del__ method in the Pool class should work. But I'm not sure we should do this.

[mattip]

It would be sufficient to modify the documentation to reflect the code.

There are other objects like file that recommend[0] calling a method to release resources without depending on implementation-specific details like garbage collection.

[0] https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files

[pitrou]

Indeed, I think this simply needs a documentation fix.

[tzickel]

What other object in the standard lib, leaks resources when deleted in CPython ? Even that documentation says the garbage collector will eventually destroy it, just like here... I think there is an implementation bug.