multiprocessing memory huge usage

[Windsooon]

BPO	34134
Nosy	@pitrou, @methane, @zware, @zhangyangyu, @Windsooon
PRs	bpo-34134: Advise to use imap or imap_unordered when handling long iterables. #8324 [3.7] bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) #11673 [3.7] bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) #11673 [3.6] bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) #11674 [3.6] bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) #11674 [3.6] bpo-34134: Advise to use imap or imap_unordered when handling long iterables. (gh-8324) #11674
Files	test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-25.12:17:21.448>
created_at = <Date 2018-07-17.04:03:50.669>
labels = ['3.7', '3.8', 'performance']
title = 'multiprocessing memory huge usage'
updated_at = <Date 2019-01-25.12:17:21.445>
user = 'https://github.com/Windsooon'

bugs.python.org fields:

activity = <Date 2019-01-25.12:17:21.445>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-25.12:17:21.448>
closer = 'pitrou'
components = []
creation = <Date 2018-07-17.04:03:50.669>
creator = 'Windson Yang'
dependencies = []
files = ['47698']
hgrepos = []
issue_num = 34134
keywords = ['patch']
message_count = 14.0
messages = ['321788', '321791', '321796', '321797', '321800', '321836', '321837', '321856', '321857', '321864', '321867', '334354', '334356', '334357']
nosy_count = 5.0
nosy_names = ['pitrou', 'methane', 'zach.ware', 'xiang.zhang', 'Windson Yang']
pr_nums = ['8324', '11673', '11673', '11674', '11674', '11674']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue34134'
versions = ['Python 3.7', 'Python 3.8']

[Windsooon]

I'm using macOX and I got huge memory usage when using generator with multiprocess. (see file)

I think this is because (https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L383)

    if not hasattr(iterable, '__len__'):
        iterable = list(iterable)

    if chunksize is None:
        chunksize, extra = divmod(len(iterable), len(self._pool) * 4)
        if extra:
            chunksize += 1

When we convert an iterable to list(iterable), we lost the advantage of using the generator. I'm not sure how to fix it, maybe we can set a default value for an object don't have '__len__' attr, any ideas?

[methane]

Do you imap or imap_unorderd?
They are intended for use with iterator, including generator.

[Windsooon]

Thank you for the hint, INADA. I think we should add something like "if you are using generator, consider use imap instead" in https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.pool.Pool.map

[methane]

I think we should add something like "if you are using generator, consider use imap instead"

I think it's not good hint.
There are short generator. And there are long (or infinite) iterator other than generator too.
Maybe, "if iterator is not sequence (e.g. generator) and can be very big, consider using imap or imap_unorderd with explicit chunksize option for better efficiency."

But I'm not good at writing English.
Someone other than me can write better paragraph.

[Windsooon]

Thank you, I will try to make a pull request and let other to edit it.

[zhangyangyu]

One thing worth a try here maybe turn len to operator.length_hint. But I am not sure it's a good idea and just a mention here.

[Windsooon]

Thank you Xiang Zhang, I found the code keeps hanging when I use imap, I will try to figure out tomorrow.

[Windsooon]

The code didn't work with imap because imap create a generator, so we can't access result outside the with statement.

    with Pool(os.cpu_count()) as p:
        result = p.imap(clean_up, k, 50)
    for r in result:
        print(r)

In https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#using-a-pool-of-workers I found the correct example. I'm not sure should me add example or warning in imap function.

[zhangyangyu]

Why accessing the result outside of with block? The pool is terminated while exiting the block before the work is done.

[Windsooon]

Yes, we should not. But we can do this when use map function. the document gives a good example but doesn't say much about real differences between map and imap. Maybe we should add some notes like INADA suggest.

map function will convert iterable to list if it doesn't implement __len__ function, so if you are using a generator, you should consider use imap. As well as add a warning about don't try to access the result outside the with statement.

But if you guys think the docs are good enough, please close this issue.

[zhangyangyu]

I'm +1 for INADA's change, but not more examples trying to distinguish every detail difference.