Failed on partial data (I think)

[llimllib]

Sorry for the rapid fire issues!

I wanted to preview my images, and I thought that -pre was -p, so I started a query:

$ felicette -p -c 34.6064 -70.345
✗ required data doesn't exist, downloading blue band
b2:  51%|█████████████████████████████████████████████████████████████████▎                                                             | 28.9M/56.2M [00:08<00:16, 1.73MB/s]b2:  52%|█████████████████████████████████████████████████████████████████▋                                                             | 29.1M/56.2M [00:08<00:07, 3.64MB/s]

Aborted!

When I looked at the docs and realized I wanted -pre instead of -p, I aborted the felicette task with ^c and ran it again, but this time it thought it had the full blue band and didn't download it, then failed on decoding a partial tiff:

$ felicette -pre -c 34.6064 -70.345
✗ preview data doesn't exist, downloading image
preview: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 162k/162k [00:00<00:00, 518kB/s]
Preview image saved at:
/Users/llimllib/felicette-data/LC81511092019265/LC81511092019265-preview.jpg
Are you sure you want to see an enhanced version of the image at the path shown above? [Y/n]
 ✓  required data exists for blue band
✗ required data doesn't exist, downloading green band
b3: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 57.7M/57.7M [00:23<00:00, 2.56MB/s]
✗ required data doesn't exist, downloading red band
b4: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 59.3M/59.3M [00:15<00:00, 4.14MB/s]
ERROR 1: TIFFFillTile:Read error at row 7168, col 7168, tile 159; got 281659 bytes, expected 310764
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: /Users/llimllib/felicette-data/LC81511092019265/LC81511092019265-b2.tiff, band 1: IReadBlock failed at X offset 15, Y offset 8: TIFFReadEncodedTile() failed.
Traceback (most recent call last):
  File "rasterio/_io.pyx", line 695, in rasterio._io.DatasetReaderBase._read
  File "rasterio/shim_rasterioex.pxi", line 142, in rasterio._shim.io_multi_band
  File "rasterio/_err.pyx", line 182, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: /Users/llimllib/felicette-data/LC81511092019265/LC81511092019265-b2.tiff, band 1: IReadBlock failed at X offset 15, Y offset 8: TIFFReadEncodedTile() failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/llimllib/.pyenv/versions/3.8.5/bin/felicette", line 8, in <module>
    sys.exit(main())
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/felicette/cli.py", line 76, in main
    process_landsat_data(data_id, bands)
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/felicette/sat_processor.py", line 153, in process_landsat_data
    process_landsat_rgb(id, bands)
  File "/Users/llimllib/.pyenv/versions/3.8.5/lib/python3.8/site-packages/felicette/sat_processor.py", line 94, in process_landsat_rgb
    b = b2.read(1)
  File "rasterio/_io.pyx", line 359, in rasterio._io.DatasetReaderBase.read
  File "rasterio/_io.pyx", line 698, in rasterio._io.DatasetReaderBase._read
rasterio.errors.RasterioIOError: Read or write failed. /Users/llimllib/felicette-data/LC81511092019265/LC81511092019265-b2.tiff, band 1: IReadBlock failed at X offset 15, Y offset 8: TIFFReadEncodedTile() failed.

I hope these issue reports are helpful and not bothersome! Thank you for putting a cool tool out into the world, I appreciate it.

edit: I would report the version but there's no felicette --version - you may want to add that!

[plant99]

Hi @llimllib , I think #4 and this issue can be combined and worked on. Will add another action item for --version.
Thanks a lot for your feedback and encouragement!

[plant99]

3447b78 this commit partially fixes this issue, by catching the RasterIOError, deleting the data dirs, and reattempting download and processing.

[plant99]

A long term solution to this would be using check-sums, but will add this to to-do, due to less bandwidth. Feel free to reopen the issue if the problem persists after the next release

[llimllib]

ah sorry I missed #4, my bad