jpeg encoder error with some combination of exif / icc_profile data

[OriHoch]

I'm getting jpeg encoder error for a specific file. The error occurs if either exif data or icc_profile are in the save params. See below for the image and full code

The bug occurs on either of these:
params.update({'icc_profile': im.info['icc_profile']})
params.update({'exif': im.info['exif']})

But it does not occur for this:
params.update({'icc_profile': im.info['icc_profile'], 'exif': im.info['exif']})

And it does not occur if no params.update is done.

The bug actually occurs in usage of the sorl thumbnail library, but I pinned it down to the following code.

In [1]: from PIL import Image, PILLOW_VERSION
In [2]: print PILLOW_VERSION
3.0.0
In [3]: im = Image.open('im.jpg')
In [4]: im.thumbnail((200,200))
In [5]: params = {'format': 'JPEG', 'quality': 95, 'optimize': 1}
In [6]: params.update({'icc_profile': im.info['icc_profile']})
In [7]: im.save('out.jpg', **params)
Suspension not allowed here
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-6-ca529ac52297> in <module>()
----> 1 im.save('out.jpg', **params)
venv/local/lib/python2.7/site-packages/PIL/Image.pyc in save(self, fp, format, **params)
   1663 
   1664         try:
-> 1665             save_handler(self, fp, filename)
   1666         finally:
   1667             # do what we can to clean up
venv/local/lib/python2.7/site-packages/PIL/JpegImagePlugin.pyc in _save(im, fp, filename)
    689     bufsize = max(ImageFile.MAXBLOCK, bufsize, len(info.get("exif", b"")) + 5)
    690 
--> 691     ImageFile._save(im, fp, [("jpeg", (0, 0)+im.size, 0, rawmode)], bufsize)
    692 
    693 
venv/local/lib/python2.7/site-packages/PIL/ImageFile.pyc in _save(im, fp, tile, bufsize)
    488             s = e.encode_to_file(fh, bufsize)
    489             if s < 0:
--> 490                 raise IOError("encoder error %d when writing image file" % s)
    491             e.cleanup()
    492     if hasattr(fp, "flush"):

IOError: encoder error -2 when writing image file

[hugovk]

Does the error occur with Pillow 2.9.0?

pip install pillow==2.9.0

[radarhere]

I'm able to reproduce in both master and 2.9.x

[wiredfool]

Just as a wild guess, try setting ImageFile.MAXBLOCK to something like 1024*1024 prior to running this.

[OriHoch]

@wiredfool - yes, it solves the problem

what are the implications of this?

in sorl library, they do this:
ImageFile.MAXBLOCK = max(ImageFile.MAXBLOCK, image.size[0] * image.size[1])

see https://github.com/mariocesar/sorl-thumbnail/blob/master/sorl/thumbnail/engines/pil_engine.py#L209

is it safe to have it changed there to have minimum of 1024*1024?

[wiredfool]

This is a long running issue.

Essentially, there are a few things that can happen that require that the JpegEncoder be able to process all of something in one shot. The exif is one of those things, It's possible that the iccprofile is another, and in some cases, the entire resulting image needs to fit in the buffer. There's a balance between allocating that buffer every single time and only allocating a larger buffer when it's necessary. MAXBLOCK is the size of the buffer, and I gave you a value to test that was big enough to diagnose the issue.

Setting the maxblock to size[0]size[1] is a rough heuristic that may succeed, but may fail for cases where Optimize is true and the Jpeg is not very compressible, since the raw size of the Jpeg is 3size[0]*size[1], since it's a three band image and it's allocating a buffer that's 1/3 of that size. I think we've found that 2_size[0]_size[1] catches those corner cases. But that's only in cases where the JpegDecoder needs to pass back the whole image in one buffer.

So it's possible that we've missed a case where the whole image needs to come back, or it's something with the metadata in that image. Since it's only if you update the info, it's likely the metadata.