image classifcation function throws error with corrupt images files

[matobler]

When an image file is corrupted (e.g. was not terminated) we get the following error message from the classifySpecies function:

`2022-03-24 14:24:32.329992: W tensorflow/core/framework/op_kernel.cc:1680] Unknown: OSError: image file is truncated (35 bytes not processed)
Traceback (most recent call last):

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call
ret = func(*args)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper
return func(*args, **kwargs)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator
for data in generator_fn():

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn
yield x[i]

File "", line 109, in getitem

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\PIL\Image.py", line 1146, in crop
self.load()

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\PIL\ImageFile.py", line 248, in load
"image file is truncated "

OSError: image file is truncated (35 bytes not processed)

Error in py_call_impl(callable, dots$args, dots$keywords) :
UnknownError: 2 root error(s) found.
(0) Unknown: OSError: image file is truncated (35 bytes not processed)
Traceback (most recent call last):

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in call
ret = func(*args)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper
return func(*args, **kwargs)

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func
values = next(generator_state.get_iterator(iterator_id))

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator
for data in generator_fn():

File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn
yield x[i]

File "", line 109,
In addition: Warning message:
In predict_generator(., generator = cropGenerator, steps = predict_steps, :
`

Need to add error handling for this.

[tkswanson]

Ive tried, its really hard to do with the image generator. I can't seem to find the right place to catch the error. I ended up just allowing truncated images in the current version...

…

On Thu, Mar 24, 2022, 15:38 Mathias Tobler ***@***.***> wrote: When an image file is corrupted (e.g. was not terminated) we get the following error message from the classifySpecies function: `2022-03-24 14:24:32.329992: W tensorflow/core/framework/op_kernel.cc:1680] Unknown: OSError: image file is truncated (35 bytes not processed) Traceback (most recent call last): File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in *call* ret = func(*args) File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper return func(*args, **kwargs) File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func values = next(generator_state.get_iterator(iterator_id)) File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator for data in generator_fn(): File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn yield x[i] File "", line 109, in *getitem* File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\PIL\Image.py", line 1146, in crop self.load() File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\PIL\ImageFile.py", line 248, in load "image file is truncated " OSError: image file is truncated (35 bytes not processed) Error in py_call_impl(callable, dots$args, dots$keywords) : UnknownError: 2 root error(s) found. (0) Unknown: OSError: image file is truncated (35 bytes not processed) Traceback (most recent call last): File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\ops\script_ops.py", line 249, in *call* ret = func(*args) File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper return func(*args, **kwargs) File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 892, in generator_py_func values = next(generator_state.get_iterator(iterator_id)) File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 822, in wrapped_generator for data in generator_fn(): File "C:\Users\mtobler\ANACON1\envs\TE1D751\lib\site-packages\keras\engine\data_adapter.py", line 948, in generator_fn yield x[i] File "", line 109, In addition: Warning message: In predict_generator(., generator = cropGenerator, steps = predict_steps, : ` Need to add error handling for this. — Reply to this email directly, view it on GitHub <#3>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBLJ4ST5JDML6R3HL4HXP3VBTVH5ANCNFSM5RSP7LZQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[matobler]

This particular issue was likely solved with the new tf.data generator, but there might still be some cases that could lead to errors. Currently I check for zero-length files and the decoder function can handle truncated files. But there might be other possible corruptions.