COGs built from VRT don't validate

[drwelby]

After building a COG from a vrt:

rio cogeo create my.vrt my.tif -p jpeg

the COG doesn't validate:

> rio cogeo validate /vsis3/bucket/my.tif
The following errors were found:
- The offset of the IFD for overview of index 1 is 884184, whereas it should be greater than the one of index 0, which is at byte 3167886
- The offset of the IFD for overview of index 2 is 299436, whereas it should be greater than the one of index 1, which is at byte 884184
- The offset of the IFD for overview of index 3 is 149834, whereas it should be greater than the one of index 2, which is at byte 299436
- The offset of the IFD for overview of index 4 is 107736, whereas it should be greater than the one of index 3, which is at byte 149834
- The offset of the IFD for overview of index 5 is 94524, whereas it should be greater than the one of index 4, which is at byte 107736
- The offset of the first block of the smallest overview should be after its IFD
/vsis3/bucket/my.tif is NOT a valid cloud optimized GeoTIFF

The VRT points to a group of 256px tiffs stored in S3.

[vincentsarago]

Thanks for the issue @drwelby , any chance you can share the vrt (+tiffs) or create dummy data ?

[drwelby]

Yes, I should be able to copy some data into a public bucket.

[drwelby]

rio cogeo create /vsis3/rio-cogeo-vrt-sample/rio-cogeo-s3.vrt ./rcv-test.tif -p jpeg

of course this works locally, so I'll have to investigate our process container to see why it generates invalid COGs.

[vincentsarago]

@drwelby could you check rasterio and gdal version ?

[drwelby]

rasterio==1.0.26
rio-cogeo==1.1.0
pygdal==2.1.2.5

libgdal20=2.1.2
libgdal-dev=2.1.2

is what is installed in the container, but I should check to make sure nothing is getting downgraded

Locally I have:

GDAL 2.4.1
rasterio 1.0.24

[vincentsarago]

@drwelby This is strange and it seems to me /vsis3/bucket/my.tif is not the output of rio cogeo create. I've just tested and it works fine on my side (even if it really slow).

To be honest, It might be faster to do

gdal_translate my.vrt my.tif -co COMPRESS=JPEG
rio cogeo create my.tif my_cog.tif -p jpeg

[drwelby]

We upgraded to GDAL 2.4.1 in the container and now valid COGs are being created.

Thanks for the suggestion of adding a gdal_translate step - we'll benchmark it and see if it provides any gains.

[drwelby]

Tried it out and writing the intermediate file turned out to be ~5% slower in our setup.

[vincentsarago]

@drwelby are we good to close here ?

[drwelby]

Closing, problem on our end.