Trouble loading a 6gb tiff extension file with shape (36608, 186496)

[Yuvi-416]

Hi there,
I couldn't load 40X magnification wsi image which has a shape = (36608, 186496), size = 6827245568 (6.83GB)). I wish I could upload an image file, but I couldn´t bcoz of file size.

What did you do?
I tried:
1.

from io import BytesIO
from PIL import Image, ImageFile

ImageFile.LOAD_TRUNCATED_IMAGES = True

with open(input_com_40, 'rb') as f:
b = BytesIO()
f.seek(15, 0)

b.write(f.read())

im = Image.open(b)
im.load()

Error:
Traceback (most recent call last):
File "/Users/yubraj/PycharmProjects/VIdeo_frame_extraction/NDPI-ext/TEMP-1-COMPRESSION.py", line 35, in 
im = Image.open(b)
File "/Users/yubraj/.conda/envs/Temp1/lib/python3.7/site-packages/PIL/Image.py", line 2944, in open
"cannot identify image file %r" % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7fca1ff39ef0>

import PIL.Image
PIL.Image.MAX_IMAGE_PIXELS = None
im = Image.open(input_com_40)
im.save(str(output_com_JPEG2000_10)+'40X_c2.j2k', irreversible=False)

Error:
Traceback (most recent call last):
File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1448, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/yubraj/PycharmProjects/VIdeo_frame_extraction/NDPI-ext/TEMP-1-COMPRESSION.py", line 42, in 
im = Image.open(input_com_40)
File "/Users/yubraj/.conda/envs/Temp1/lib/python3.7/site-packages/PIL/Image.py", line 2944, in open
"cannot identify image file %r" % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file '/Users/yubraj/OneDrive/40X-c-0.tif'

I can load file with tifffile package.

img = tifffile.imread(input_com_40)

Output:
Shown in image.

What are your OS, Python and Pillow versions?
OS: Catalina 10.15..
Python: 3.7
Pillow: 8.0(latest)

I would like to load this image and then wants to compress it with jpeg2000.

Any suggestion will be highly appreciated.

[hugovk]

You could uncomment the logging line here to get a bit more info:

Pillow/src/PIL/Image.py

Lines 2918 to 2922 in 76d0951

	except (SyntaxError, IndexError, TypeError, struct.error):
	# Leave disabled by default, spams the logs with image
	# opening failures that are entirely expected.
	# logger.debug("", exc_info=True)
	continue

[Yuvi-416]

I tried your suggestion, but I am still getting the same error as above.
Do you think I am getting this error bcoz of the image size?

[hugovk]

The same error is expected, but should have extra logging output. Did you get no extra output? I guess logging would need enabling, or try this instead:

 except (SyntaxError, IndexError, TypeError, struct.error) as e: 
     # Leave disabled by default, spams the logs with image 
     # opening failures that are entirely expected. 
     # logger.debug("", exc_info=True) 
     print(e)
     continue 

It's quite possible it's due to the image size.

[Yuvi-416]

Hi there,
I again try to run it with your suggestion, but the error is still the same. I would like to send you an image file, but u have to run below code to extract three images (each image size is 6.83GB) bcoz images are stacked. I want to open and compress each extracted images. (image link: https://uapt33090-my.sharepoint.com/:i:/g/personal/ygupta_ua_pt/EYpxnbw4ydpAkEJxAolaEb4B9IJ5In-G6pj2Bq6bQvGPZg?e=tGu8qn)
Code:

img = tifffile.imread(input_com_40)
for c in range(img.shape[2]):
      cslice = img[:, :, c]
      tifffile.imwrite(str(output_com_40)+f'40X_c{c}.tif', slice)

And I am still getting the same error without any further error message.

Error:
Connected to pydev debugger (build 202.7660.27)
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/yubraj/PycharmProjects/VIdeo_frame_extraction/NDPI-ext/TEMP-1-COMPRESSION.py", line 43, in <module>
    im = PIL.Image.open(input_com_40, formats=['TIFF'])
  File "/Users/yubraj/.conda/envs/Temp1/lib/python3.7/site-packages/PIL/Image.py", line 2945, in open
    "cannot identify image file %r" % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file '/Users/yubraj/40X_c0.tif'

Process finished with exit code 1

Please take a look.

[hugovk]

import tifffile  # python3.8 -m pip install tifffile

input_com_40 = "01_x40_z0.tif"
output_com_40 = "out_"

img = tifffile.imread(input_com_40)
for c in range(img.shape[2]):
      cslice = img[:, :, c]
      tifffile.imwrite(str(output_com_40)+f'40X_c{c}.tif', slice)

With this, I got this after 3.5 minutes:

Traceback (most recent call last):
  File "1.py", line 9, in <module>
    tifffile.imwrite(str(output_com_40)+f'40X_c{c}.tif', slice)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tifffile/tifffile.py", line 787, in imwrite
    return tif.write(data, shape, dtype, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tifffile/tifffile.py", line 1296, in write
    data = numpy.asarray(data, datadtype, 'C')
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: float() argument must be a string or a number, not 'type'

It had used up (and then freed) at least 8 GB of disk space in the process. Maybe it ran out of space, I'm a bit low.

Another debugging idea, temporarily delete the except (SyntaxError, IndexError, TypeError, struct.error) as e: block and let it raise the original exception, and we can see the full traceback.

[Yuvi-416]

Yeah, I agree with you. To extract that image u need at least 10 GB per image.
I try your suggestion by commenting the line which you have refer to me:
``
elif result:
fp.seek(0)
im = factory(fp, filename)
_decompression_bomb_check(im.size)
return im
# except (SyntaxError, IndexError, TypeError, struct.error) as e:
# Leave disabled by default, spams the logs with image
# opening failures that are entirely expected.
# logger.debug("", exc_info=True)
# print(e)
# continue
´´´
But still, I didn´t receive any further or extra error message beside the last one.
ERROR:
Traceback (most recent call last):
File "/Users/yubraj/PycharmProjects/VIdeo_frame_extraction/NDPI-ext/TEMP-1-COMPRESSION.py", line 43, in
im = PIL.Image.open(input_com_40, formats=['TIFF'])
File "/Users/yubraj/.conda/envs/Temp1/lib/python3.7/site-packages/PIL/Image.py", line 2945, in open
"cannot identify image file %r" % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file '/Users/OneDrive/1-slice/40X_c0.tif'

Process finished with exit code 1

[cgohlke]

each image size is 6.83GB
PIL.UnidentifiedImageError: cannot identify image file '/Users/yubraj/40X_c0.tif'

The extracted images are in BigTIFF format, which is not supported by Pillow.

[hugovk]

Thanks for checking!

Let's close this as a duplicate of #4513.