Exception while reading EXIF data from JPEG

[andy-z]

There is one weird image that causes Image.open() to generate exception on 32-bit Python on windows. It looks like pillow thinks that EXIF data is corrupted though other applications can read exif data normally.

What did you do?

Call PIL.Image.open("file.jpg")

What did you expect to happen?

Call to return normally without exception or complaints

What actually happened?

Pillow generates multiple UserWarning messages about possibly corrupt EXIF, and it also raises exception on 32-bit Python build.

What versions of Pillow and Python are you using?

Pillow 5.2.0, Python 3.6

Here is what happens on 32-bit Windows build:

(venv-win-py36) PS M:\> python
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
>>> Image.open('046.JPG')
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1441814 bytes but only got 0. Skipping tag 33
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 428 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 196608 bytes but only got 0. Skipping tag 60
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 196608 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 524288 bytes but only got 0. Skipping tag 240
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1048576 bytes but only got 0. Skipping tag 13222
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 2097152 bytes but only got 0. Skipping tag 1860
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 458752 bytes but only got 0. Skipping tag 10869
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 458752 bytes but only got 0. Skipping tag 1821
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 393216 bytes but only got 0. Skipping tag 1024
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 786432 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1572864 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 33752069 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "M:\venv-win-py36\lib\site-packages\PIL\Image.py", line 2609, in open
    im = _open_core(fp, filename, prefix)
  File "M:\venv-win-py36\lib\site-packages\PIL\Image.py", line 2599, in _open_core
    im = factory(fp, filename)
  File "M:\venv-win-py36\lib\site-packages\PIL\JpegImagePlugin.py", line 780, in jpeg_factory
    im = JpegImageFile(fp, filename)
  File "M:\venv-win-py36\lib\site-packages\PIL\ImageFile.py", line 102, in __init__
    self._open()
  File "M:\venv-win-py36\lib\site-packages\PIL\JpegImagePlugin.py", line 339, in _open
    handler(self, i)
  File "M:\venv-win-py36\lib\site-packages\PIL\JpegImagePlugin.py", line 124, in APP
    exif = self._getexif()
  File "M:\venv-win-py36\lib\site-packages\PIL\JpegImagePlugin.py", line 432, in _getexif
    return _getexif(self)
  File "M:\venv-win-py36\lib\site-packages\PIL\JpegImagePlugin.py", line 467, in _getexif
    info.load(file)
  File "M:\venv-win-py36\lib\site-packages\PIL\TiffImagePlugin.py", line 738, in load
    fp.seek(offset)
OverflowError: Python int too large to convert to C ssize_t
>>> import PIL
>>> PIL.__version__
'5.2.0'

On 64-bit Linux build I see the same warnings, but exception is not raised:

(venv-3.6) @ubu-srv-vm:~$ python
Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
>>> Image.open("046.JPG")
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1441814 bytes but only got 0. Skipping tag 33
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 428 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 196608 bytes but only got 0. Skipping tag 60
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 196608 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 524288 bytes but only got 0. Skipping tag 240
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1048576 bytes but only got 0. Skipping tag 13222
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 2097152 bytes but only got 0. Skipping tag 1860
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 458752 bytes but only got 0. Skipping tag 10869
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 458752 bytes but only got 0. Skipping tag 1821
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 393216 bytes but only got 0. Skipping tag 1024
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 786432 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1572864 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 33752069 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 24113876360 bytes but only got 0. Skipping tag 30719
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 30889119128 bytes but only got 0. Skipping tag 62463
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 238688098 bytes but only got 0. Skipping tag 63316
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 282 bytes but only got 0. Skipping tag 1
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:747: UserWarning: Possibly corrupt EXIF data.  Expecting to read 513 bytes but only got 0. Skipping tag 1
  " Skipping tag %s" % (size, len(data), tag))
/home/user/venv-3.6/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:764: UserWarning: Corrupt EXIF data.  Expecting to read 12 bytes but only got 4.
  warnings.warn(str(msg))
<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2560x1920 at 0x7F5CEF8DF080>
>>> import PIL
>>> PIL.__version__
'5.2.0'

I am attaching image file but that file has actual image section stripped (the whole 0xFFDA section removed, but rest of the content is intact) as this is not my photo and I don't want to upload someone else's pictures here. With this stripped image I can still see warnings and exception on 32-bit build:

[andy-z]

I think the reason for that is slightly different header which is not processed by Pillow correctly. Here is the TIFF header ("that follows "Exif\0\0" in EXIF section) from a file that can be read by pillow:

$ od -Ax -v -j 12 -to1 ../ok.jpg | head -1
00000c 115 115 000 052 000 000 000 010 000 013 001 017 000 002 000 000

and here the header of the file which causes troubles:

$ od -Ax -v -j 12 -to1 ../not-ok.jpg | head -1
00000c 115 115 000 052 000 000 000 012 132 132 000 013 001 017 000 002

The difference is two extra bytes (132 132) that follow header in latter case. The offset in latter case (012) takes into account those two bytes (normal offset is 010), but apparently Pillow does not take into account that offset when parsing remaining TIFF structure.

I am not sure what is the best way to fix this, I looked at ImageFileDirectory_v2.load() method and it looks like it makes assumptions about current file offset and I'm not sure that fixing this won't break the assumptions. I tried trivial fix by adding fp.seek(self.next) call before parsing and that seems to fix the problem for my file:

712         self.reset()
713         self._offset = fp.tell()
714         fp.seek(self.next)

Let me know if you want me to open PR with this fix.

[radarhere]

I have created PR #3584 to resolve this. As you are the only one with the actual image, it would be good if you could test that branch against your image, on Windows and on Linux.

[andy-z]

Thanks for the fix! I have built radarhere:ifd_offset on Linux and it works fine. I do not know how to build it for Windows platform from sources, though I can certainly test it if I could do binary pip install from somewhere.

[hugovk]

Good to hear!

To test on Windows, you can find and edit JpegImagePlugin.py (eg. M:\venv-win-py36\lib\site-packages\PIL\JpegImagePlugin.py) and add in the two new lines:

https://github.com/python-pillow/Pillow/pull/3584/files

[andy-z]

With that small patch it works fine on windows too (applied on top of Pillow 5.4.1).

[radarhere]

Excellent. Thanks very much