New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10 #72692
Comments
Multiprocessing is throwing this error when dealing with large amounts of data (all floating points an integers), but none of which exceeds the number boundaries in the error that it throws: File "/root/anaconda3/lib/python3.5/multiprocessing/pool.py", line 268, in starmap
It works fine on any number of subsets of this data, but not when put together. |
This has nothing to do with the _values_ you're passing - it has to do with the length of the pickle string: def _send_bytes(self, buf):
n = len(buf)
# For wire compatibility with 3.2 and lower
header = struct.pack("!i", n) IT'S BLOWING UP HERE
if n > 16384:
...
self._send(header)
self._send(buf) where the traceback shows it's called here: self._send_bytes(ForkingPickler.dumps(obj)) Of course the less data you're passing, the smaller the pickle, and that's why it doesn't blow up if you pass subsets of the data. I'd suggest rethinking how you're sharing data, as pushing two-gigabyte pickle strings around is bound to be the least efficient way possible even if it didn't blow up ;-) |
This looks as a duplicate of bpo-17560. |
Ah, should have picked that up, coding at 3:30am doesn't do wonders for Thanks Tim, I'll keep that in mind! *Justin Ting* On Sun, Oct 23, 2016 at 3:48 AM, Tim Peters <report@bugs.python.org> wrote:
|
Actually, on further inspection, I seem to be having a slightly different problem with the same error that I initially described now. Even after modifying my code so that each python forked off to another process was only given the following arguments: where models_shape, start, and end are only single integers, the same error still comes up as a result. Within each process, I'm reading in a (relatively small, only 12MB) .npy ndarray and taking the [start:end] slice. |
Closed as a duplicate of bpo-17560. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: