How to load new data uploaded to S3 if it previously failed?

[christianaaronschroeder]

I'm using titiler.application installed through pip and am trying to solve for a specific use case. That being when a user tries to load a COG from S3, finds out it isn't there, uploads the file, and then tries to access it again.

Right now if I try to access a non-existent file with http://0.0.0.0:8000/cog/info?url=s3://my-bucket/my_file.tif, I get the expected response,

{"detail":"The specified key does not exist."}

So I go and upload the file to my S3 bucket and try http://0.0.0.0:8000/cog/info?url=s3://my-bucket/my_file.tif again. Now I would expect to get a proper response with the file's info, but instead I get this,

{"detail":"'/vsis3/my-bucket/my_file.tif' does not exist in the file system, and is not recognized as a supported dataset name."}

Also this is a valid file. If I have it uploaded before trying to access it, it will work.

Does anyone know what is causing this and maybe a way around it?

[vincentsarago]

@christianaaronschroeder the problem, I think, is due to GDAL Cache. Sadly I don't think there is an easy fix for this 😭

some (maybe) relevant tickets:

• Failure to retry a read for an s3-COG (GeoTIFF on AWS s3) rasterio/rasterio#2119
• Occasional "not recognized as a supported file format" errors when reading from S3 rasterio/rasterio#1686 (comment)

[christianaaronschroeder]

Thanks!! The second link led me to the right config. Setting CPL_VSIL_CURL_NON_CACHED=/vsis3/my-bucket worked.

[mradamcox]

Was just linking this one too: #561 about the same thing.