OverflowError: size does not fit in an int #2251

Open
ziky90 opened this Issue Nov 24, 2016 · 3 comments

Projects

None yet

2 participants

@ziky90
ziky90 commented Nov 24, 2016 edited

Hi,

I am getting an error very similar to #1475 with the Pillow==3.4.2, that is supposed to be fixed.

Actually I'm getting:

File "/home/jan/my_script.py", line 230, in save
    img = Image.fromarray(image, 'RGB')
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2187, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2137, in frombuffer
    return frombytes(mode, size, data, decoder_name, args)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2070, in frombytes
    im.frombytes(data, decoder_name, args)
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 741, in frombytes
    s = d.decode(data)
OverflowError: size does not fit in an int

Details about my image are following:
shape = (31000, 30030, 3)
dtype = uint8

My Python version is Python 2.7.12

@wiredfool
Member

Can you post a minimal script that reproduces the error?

@ziky90
ziky90 commented Nov 25, 2016 edited

@wiredfool Here is the minimal script that reproduces the error:

from PIL import Image
import  numpy as np

image = np.random.randint(255, size=(31000, 30030, 3))
image = image.astype('uint8')

# next line will cause the OverflowError
img = Image.fromarray(image, 'RGB')
@wiredfool
Member
wiredfool commented Nov 25, 2016 edited

The underlying issue in #1475 isn't actually fixed, Image.frombuffer is limited to a 2gb chunk. I think even with a mmap-able image (e.g. RGBX or RGBA), we'd still be running into an integer overflow issue in the raw decoder.

I kept getting memory errors on the randint line, but running this as:

from PIL import Image
import  numpy as np

a = np.zeros((31000, 30030, 3), dtype=np.uint8)

# next line will cause the OverflowError
img = Image.fromarray(a, 'RGB')

Gives me the same issue, it's failing in decode.c:127, the same spot as in #1475.

And attempting to mmap this:

a = np.zeros((31000, 30030, 4), dtype=np.uint8)
img = Image.fromarray(a, 'RGBX')

Fails in a new and interesting way:

  File "numpy_overflow.py", line 12, in <module>
    img = Image.fromarray(a, 'RGBX')
  File "/home/erics/vpy27-dbg/local/lib/python2.7/site-packages/Pillow-3.4.2-py2.7-linux-x86_64.egg/PIL/Image.py", line 2187, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/home/erics/vpy27-dbg/local/lib/python2.7/site-packages/Pillow-3.4.2-py2.7-linux-x86_64.egg/PIL/Image.py", line 2132, in frombuffer
    core.map_buffer(data, size, decoder_name, None, 0, args)
MemoryError: Integer overflow in ysize
[84908 refs]

which is in this code: map.c:345

    if (!PyArg_ParseTuple(args, "O(ii)sOn(sii)", &target, &xsize, &ysize,
                          &codec, &bbox, &offset, &mode, &stride, &ystep))
        return NULL;

    if (!PyImaging_CheckBuffer(target)) {
        PyErr_SetString(PyExc_TypeError, "expected string or buffer");
        return NULL;
    }

    if (stride <= 0) {
        if (!strcmp(mode, "L") || !strcmp(mode, "P"))
            stride = xsize;
        else if (!strncmp(mode, "I;16", 4))
            stride = xsize * 2;
        else
            stride = xsize * 4;
    }

    if (ysize > INT_MAX / stride) {
        PyErr_SetString(PyExc_MemoryError, "Integer overflow in ysize");
        return NULL;
    }
@wiredfool wiredfool added the Duplicate label Nov 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment