Fail-fast or skipping of problematic images

[tokee]

I am in the process of testing pix-plot with 26K images with varying names.

Some of them contained parentheses, which caused pix-plot to fail very late in the process. Others had an apostrophe, which gave the same error.

Traceback (most recent call last):
  File "pix-plot/utils/process_images.py", line 349, in <module>
    PixPlot(image_dir=sys.argv[1], output_dir='output')
  File "pix-plot/utils/process_images.py", line 58, in __init__
    self.write_json()
  File "pix-plot/utils/process_images.py", line 261, in write_json
    'positions': self.get_2d_image_positions(),
  File "pix-plot/utils/process_images.py", line 174, in get_2d_image_positions
    return self.get_image_positions(model)
  File "pix-plot/utils/process_images.py", line 218, in get_image_positions
    with Image.open(thumb_path) as image:
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2410, in open
    fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: "output/thumbs/32px/bi_20150530-1609_2_AstaCykelSt'tteben.jpg"

I checked and the file output/thumbs/32px/bi_20150530-1609_2_AstaCykelSt'tteben.jpg does exist, but I can understand why it could give problems in a script.

Some images were corrupt, which (guessing here) meant that the thumbnail-processing left no file, which again caused some later step to fail with a different error. Unfortunately I did not keep that error message.

So far I have handled the problem by finding & removing files with problematic characters when the script throws an exception and removing all corrupt images using for F in sources_sshfs/*.jpg; do if [[ -z $(identify "$F" 2> /dev/null) ]]; then echo "Removing $F" ; rm "$F" ; fi ; done, but I haven't completed a full run of the 26K images yet.

Whether due to naming or format problems, the overall problem is the same: The script needs to run for 10+ minutes before it fails. Preferably it could be made to just skip the problematic images, but alternatively it would be nice if it could fail early in the process?

[duhaime]

Thanks for reaching out on this, and sorry for your troubles @tokee. I agree that failing fast is nice, though skipping the problematic images could work as well.

I'm curious about the corrupt images--were the inputs corrupt, or the outputs, or both? Could you send a few example images that produced corrupt outputs? If so, those samples could help determine how best to treat some cases...

[tokee]

It was definitely the input images that were corrupt. I could dig up some samples, but as ImageMagic's own identify fails on them I assume it is a simple case of bit rot on my spinning rust (I say "simple" because I have backups).

[duhaime]

@tokee I just pushed up a branch validate-inputs -- could you try using that branch to see if this resolves your troubles?

[tokee]

Looks good! It catches both parentheses and spaces in filenames. Thanks!

[duhaime]

Awesome, thanks for following up!