Silently fails to process gallery images

[rogerbinns]

On my first machine, everything works perfectly. I have several galleries and they get processed fine.

On a second machine all the files in galleries are listed as being processed (render_galleries:output/galleries/ALBUM/filename) for the image as well as its thumbnail. However the output/galleries/ALBUM directories do not have most of the photos. Sometimes there are none (out of ~10) and in one 2 out 4. I can't see any pattern to why.

Some dependencies failed to build because zlib dev was not present, but I fixed that. Pillow says both JPEG and PNG (the two formats I use) have support available.

[rogerbinns]

I diagnosed this. If Pillow if being used then you get the silent failures. If (Ubuntu 13.04) python-imaging is installed then it works fine.

[ralsina]

That sounds like a Pillow setup failure or a Pillow bug. Nikola supports both PIL and Pillow but suggests Pillow because it's python 3 compatible and works on virtualenv.

[rogerbinns]

The big problem is the silent part. Perhaps add some code after the processing to verify the output file actually exists and is not zero length?

[ralsina]

Yep. Adding it to my task list.

[rogerbinns]

I can definitely confirm this again with 5.5.1 on two different machines. pip install --upgrade to get 5.5.1 resulted in Pillow being installed even though python-imaging is installed. And then the galleries break.

It looks like this will affect everyone running Ubuntu 13.04 (no idea about other versions)

[ralsina]

On 09/08/13 01:56, Roger Binns wrote:

I can definitely confirm this again with 5.5.1 on two different
machines. pip install --upgrade to get 5.5.1 resulted in Pillow being
installed even though python-imaging is installed. And then the
galleries break.

It looks like this will affect everyone running Ubuntu 13.04 (no idea
about other versions)

Can you mail me privately one of the imaegs that fail?

[ralsina]

I am now reproducing this. I am cleaning up the gallery plugin as part of creating the new galleries, so I should get a fix there.

[ralsina]

I have tested this with the image provided by @rogerbinns and it works so, not able to reproduce it yet.

There was one place where things could fail silently: the thumbnailing code. In those cases where it failed, it would save the original-size image to the thumbnail. I am now adding a warning there.

The only other exception catching in place is EXIF reading, which is ignored because it can still work.

Other than that, I can't see any place in the gallery code where things fail silently at all.

I am keeping this open but at this point I can't do much

[rogerbinns]

No longer silent with 6.0.1

.  render_galleries:output/galleries/2011 Coldwater Classic/2_They also aren't competing..thumbnail.jpg
WARNING: can't thumbnail galleries/2011 Coldwater Classic/2_They also aren't competing..jpg.  render_galleries:output/galleries/2011 Coldwater Classic/2_They also aren't competing..jpg
(for every image in the gallery)

########################################
TaskError - taskid:render_galleries:output/galleries/2011 Coldwater Classic/index.html
PythonAction Error
Traceback (most recent call last):
  File "/home/rogerb/.local/lib/python2.7/site-packages/doit/action.py", line 324, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/home/rogerb/.local/lib/python2.7/site-packages/nikola/plugins/task/galleries.py", line 302, in render_gallery_index
    im = Image.open(os.path.join(d_name, thumb_name))
  File "/home/rogerb/.local/lib/python2.7/site-packages/PIL/Image.py", line 1976, in open
    fp = builtins.open(fp, "rb")
IOError: [Errno 2] No such file or directory: u"output/galleries/2011 Coldwater Classic/2_They also aren't competing..thumbnail.jpg"

[ralsina]

Interesting, that means it has not generated the thumbnail, which means it may be a task ordering issue.

What does this do?

nikola build "output/galleries/2011 Coldwater Classic/2_They also aren't competing..thumbnail.jpg"

If that succeeds, does the error when doing nikola build change?

[rogerbinns]

It isn't a task ordering issue - it is a PIL/Pillow issue. If I use the python-imaging module that is standard with Ubuntu 13.04 then everything works fine. If I let pip install pillow (pip install --user --upgrade https://github.com/getnikola/nikola/archive/master.zip) then what it installs doesn't work. To get things working again I delete the Pillow that pip installed in site-packages (Pillow 2.1.0).

I deleted output/galleries and then did:

$ nikola build "output/galleries/2011 Coldwater Classic/2_They also aren't competing..thumbnail.jpg"
Scanning posts....done!
.  render_galleries:output/galleries/2011 Coldwater Classic/2_They also aren't competing..thumbnail.jpg
WARNING: can't thumbnail galleries/2011 Coldwater Classic/2_They also aren't competing..jpg$

Note the last dollar is my prompt due to lack of trailing newline in the WARNING.

[ralsina]

Ok, so it's a bug or two and I know how to hadle it now ;-)

• The thumbnail generation is not failing correctly
• The JSON generation for the gallery should depend on the thumbnail and doesn't

[ralsina]

The fix is 2 parts:

1. Try much harder to generate a thumbnail, even just copying the original file as one if nothing else works
2. Since to generate the HTML we need to read the thumbnails for size info, added the thumbnails as build dependencies for the gallery indexes

This should make the worse symptoms go away, except that your may get really really heavy thumbs in some cases if your PIL is broken.