Reading one band from a tiff stored on S3 takes more time than reading the whole file

[thmslmr]

Hello 👋

First, I'm quite new to geotiff handling with rasterio, I might be missing something obvious 😅 .

The application I'm currently working on reads data from a geotiff file stored on S3, within which one can find a lot of bands (basically image time series - one band per date). I'm trying to optimize the reading operation and I found out that reading one band at a time (e.g rasterio.open(S3_URI).read(10)) takes significantly more time than reading the whole file at once (e.g rasterio.open(S3_URI).read()). In my case, this seems to be true whether the file contains only one band, or ~150.

Unfortunately I cannot share the tiff files I'm working on.
rasterio.__version__: 1.3.7

[sgillies]

@thmslmr this is a good question about a complex subject!

When given a URI, rasterio.open() will use one of GDAL's HTTP virtual file system handlers. Instead of downloading the entire resource to temporary storage these make HTTP Range requests for chunks of the resource. When the resources are structured in a predictable way, this allows efficient access to slices of the resource.

Are your geotiffs of the COG kind? COGs are optimized for spatial slicing, that is to say getting all bands for some sub-regions, and you might expect some less great performance for slicing out a single band. In the worst case scenario, we effectively download all bands of a tiff in order to get only one band, and use more small requests than necessary to do it.

There's a link to a COG diagnostic tool in https://www.cogeo.org/developers-guide.html. You may want to run that on your tiffs and see what it reports.