Multiprocessing Send to Manager Fails for Large Payload

[maxehr]

BPO	30685
Nosy	@pitrou, @applio, @ZackerySpytz
Superseder	bpo-17560: problem using multiprocessing with really big objects?

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-07-25.15:37:14.225>
created_at = <Date 2017-06-16.16:12:46.539>
labels = ['3.7', 'type-feature', 'library']
title = 'Multiprocessing Send to Manager Fails for Large Payload'
updated_at = <Date 2019-07-25.15:37:14.197>
user = 'https://bugs.python.org/maxehr'

bugs.python.org fields:

activity = <Date 2019-07-25.15:37:14.197>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2019-07-25.15:37:14.225>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2017-06-16.16:12:46.539>
creator = 'maxehr'
dependencies = []
files = []
hgrepos = []
issue_num = 30685
keywords = []
message_count = 4.0
messages = ['296210', '297210', '348415', '348439']
nosy_count = 4.0
nosy_names = ['pitrou', 'davin', 'maxehr', 'ZackerySpytz']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '17560'
type = 'enhancement'
url = 'https://bugs.python.org/issue30685'
versions = ['Python 3.7']

[maxehr]

On line 393 of multiprocessing/connection.py, the size of the payload to be sent is serialized as an integer. This fails for sending large payloads. It should probably be serialized as a long or better yet a long long.

[pitrou]

As the comment just above that line says, this is for wire compatibility with older Python versions.

But we could have an escape hatch: a special value of the length (which happens to be encoded as a signed integer), such as -64, which would signal a following 64-bit length field. Would you like to submit a PR?

[ZackerySpytz]

It seems that this was fixed in bpo-17560.

[pitrou]

That's right. Thanks Zackery!