JPEG: Automatically check and repair broken/invalid images

[lastzero]

Go happily refuses to open JPEGs if they contain any glitches. As a user with such files, you still want to index and view them. It would therefore make sense to automate the repair of JPEGs, especially since the process is always the same.

Step 1: Use GraphicsMagick to check a JPEG for issues

GraphicsMagick is a great tool for this task. Depending on what operating system you are using, you may need to install it first: http://www.graphicsmagick.org/identify.html

Now run this command and check the report for any problems:

gm identify -verbose broken.jpg

Note that ImageMagick may also have a command for this. This describes the specific process I used and that worked for me. There are variants and it can likely be optimized so that fewer tools and libraries need to be installed.

Step 2: Use the ImageMagick convert command to create a valid copy of the image

To create a valid copy of the original JPEG:

convert broken.jpg fixed.jpg

Any glitches should then be fixed i.e. the repaired copy fixed.jpg can be opened without problems.

Related Issues:

• Failing to add jpg files #2462
• image/jpeg: unexpected EOF when decoding image file golang/go#45902
• image/jpeg: "bad RST marker" error when decoding golang/go#40130

[pbek]

Thank you very much! Even doing that manually helped me a lot with errors like import: failed creating thumbnails for 2022/08/20220820_141434_F993F0A5.jpg (EOF while decoding).
gm said gm identify: Invalid SOS parameters for sequential JPEG (20220820_141434_3172D2BD.jpg). and I could repair those images with convert.

It would be a great joy for me if PhotoPrism could do that automatically! 😸

[Lash-L]

Would be willing to take a stab at this (no promises though). Are there any examples of some broken/invalid jpegs I could use for testing/debugging?

[pbek]

Certainly there are! Thank you very much for giving it a try!

Here are my two (rather large) samples:
https://cloud.tugraz.at/index.php/s/kD2AJpyqtmQYZtT

[krystiancharubin]

@lastzero looking at the database tables it seems like we should be able to identify all the broken images by looking at files table and checking for non empty file_error.
Has something changed since may because I am not seeing entries for new panoramas that do not show up in the ui.
There is also the errors table, but we would need to parse the error message to determine the file location
import: failed creating thumbnails for 2022/09/20220906_170826_0582F88A.jpg (EOF while decoding)

[lastzero]

It is possible that files that cannot be read at all and are not yet indexed will be skipped. Accordingly, there is no entry for them in the files table.

[AlD]

How about something like #2721?

Could of course be made configurable.