[OverflowError] size doesn't fit in an int

[ajdroid]

I'm trying to open a Hamamatsu style image, which is about 500MB compressed so it is quite big but should fit in RAM (32 GB). I get Overflow error : size does not fit in an int when I try to open at levels 0, 1 or 2 but can open the slide images on all higher levels.
Any pointers on how to solve this or is it simply not possible with the python bindings?

[bgilbert]

It should be possible. OpenSlide is designed for large files, and doesn't need to load the entire slide into RAM.

Could you post a backtrace and a few lines of code leading up to the error? If the problem only occurs with a particular slide, it'd also be helpful if you could run openslide-show-properties on the slide file and post the output.

[ajdroid]

Code:

im61 = openslide.open_slide('data/61_HER2.ndpi')
level = 2
l2 = im61.read_region((0,0), level, im61.level_dimensions[level])

The error happens with multiple large slides.

Backtrace:

OverflowError Traceback (most recent call last)
in ()
1 im61 = openslide.open_slide('data/61_HER2.ndpi')
2 level = 2
----> 3 l2 = im61.read_region((0,0), level, im61.level_dimensions[level])
4 plt.imshow(l2)
5 plt.show()

/usr/lib/python2.7/dist-packages/openslide/init.pyc in read_region(self, location, level, size)
220 function is not premultiplied."""
221 return lowlevel.read_region(self._osr, location[0], location[1],
--> 222 level, size[0], size[1])
223
224

/usr/lib/python2.7/dist-packages/openslide/lowlevel.pyc in read_region(slide, x, y, level, w, h)
212 buf = (w * h * c_uint32)()
213 _read_region(slide, buf, x, y, level, w, h)
--> 214 return _load_image(buf, (w, h))
215
216 get_error = _func('openslide_get_error', c_char_p, [_OpenSlide], _check_string)

/usr/lib/python2.7/dist-packages/openslide/lowlevel.pyc in _load_image(buf, size)
167 rawmode = (sys.byteorder == 'little') and 'BGRA' or 'ARGB'
168 buf = PIL.Image.frombuffer('RGBA', size, buf, 'raw', rawmode, 0,
--> 169 1).tostring()
170 # Now load the image as RGBA, undoing premultiplication
171 return PIL.Image.frombuffer('RGBA', size, buf, 'raw', 'RGBa', 0, 1)

/usr/lib/python2.7/dist-packages/PIL/Image.pyc in frombuffer(mode, size, data, decoder_name, *args)
1897 return im
1898
-> 1899 return frombytes(mode, size, data, decoder_name, args)
1900
1901

/usr/lib/python2.7/dist-packages/PIL/Image.pyc in frombytes(mode, size, data, decoder_name, *args)
1823
1824 im = new(mode, size)
-> 1825 im.frombytes(data, decoder_name, args)
1826 return im
1827

/usr/lib/python2.7/dist-packages/PIL/Image.pyc in frombytes(self, data, decoder_name, *args)
611 d = _getdecoder(self.mode, decoder_name, args)
612 d.setimage(self.im)
--> 613 s = d.decode(data)
614
615 if s[0] >= 0:

OverflowError: size does not fit in an int

[bgilbert]

Ah! I hadn't realized you were reading an entire slide level into a PIL.Image. It seems that the problem is a Pillow bug, python-pillow/Pillow#1475, which will occur if you read more than 2²⁹ pixels in one read_region() call.

Probably the easiest workaround is to upgrade to OpenSlide Python 1.1.0, which adds a fast path for unpremultiplication that happens to avoid the Pillow bug. Another approach is to read the level in smaller chunks and then paste() the chunks into a new PIL.Image.