too small type for struct.pack/unpack in mutliprocessing.Connection

[ahcub]

BPO	35152
Nosy	@tirkarthi, @ahcub
PRs	bpo-17560: Too small type for struct.pack/unpack in mutliprocessing.Connection #10305
Superseder	bpo-17560: problem using multiprocessing with really big objects?
Files	error_on_bug_multiprocessing_output.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-03.08:21:36.329>
created_at = <Date 2018-11-03.08:00:06.191>
labels = ['3.8', '3.7', 'library', 'type-crash']
title = 'too small type for struct.pack/unpack in mutliprocessing.Connection'
updated_at = <Date 2018-11-03.08:21:36.328>
user = 'https://github.com/ahcub'

bugs.python.org fields:

activity = <Date 2018-11-03.08:21:36.328>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-03.08:21:36.329>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2018-11-03.08:00:06.191>
creator = 'ahcub'
dependencies = []
files = ['47903']
hgrepos = []
issue_num = 35152
keywords = ['patch']
message_count = 2.0
messages = ['329189', '329190']
nosy_count = 2.0
nosy_names = ['xtreak', 'ahcub']
pr_nums = ['10305']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '17560'
type = 'crash'
url = 'https://bugs.python.org/issue35152'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

[ahcub]

the problem is reproduced on big multiprocessing.Process output
when the size of the output gets bigger than signed int a struct error is raised

python3 test.py
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.5/multiprocessing/pool.py", line 125, in worker
    put((job, i, result))
  File "/usr/lib64/python3.5/multiprocessing/queues.py", line 347, in put
    self._writer.send_bytes(obj)
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.5/multiprocessing/process.py", line 252, in _bootstrap
    self.run()
  File "/usr/lib64/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.5/multiprocessing/pool.py", line 130, in worker
    put((job, i, (False, wrapped)))
  File "/usr/lib64/python3.5/multiprocessing/queues.py", line 347, in put
    self._writer.send_bytes(obj)
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

this error can be fixed by changing types that are used herу for struct.pack/unpack to !Q

[tirkarthi]

Thanks for the report and PR. I think these are related issues : bpo-17560, bpo-28506 with bpo-17560 being open one. Correct me if I am wrong here.