VSI_CACHE causing bad range requests in some circumstance (/vsizip//vsis3/)

[olsen232]

What is the bug?

I have a 4211788 byte (4MiB) zip file containing a ~10MiB tiff file in an S3 bucket.
I'm using gdal from python to open the last tiff file in the zip file.
The following works perfectly for me:
gdal.Open("/vsizip//vsis3/cdn-misc.kx.gd/foss-tickets/file_example_TIFF_10MB.zip", gdal.GA_ReadOnly)
However if I first turn on the VSI cache with gdal.SetConfigOption("VSI_CACHE", "TRUE")
Then it fails with:
S3: Request at offset 4227072, after end of file
Note that this is after the end of the file, and that it corresponds to the start of the 129th cache block - VSI cache blocks default to 32KiB in size.
So, it looks like the VSI cache is for some reason requesting a block outside the file.

I found out this works with /vsizip//vsicurl/ too, not just /vsizip/vsis3/

Steps to reproduce the issue

I've only figured out how to show this from python so far.

from osgeo import gdal

gdal.UseExceptions()

opts = {
    'VSI_CACHE': 'TRUE',
    'CPL_DEBUG': 'ON',
}

for k, v in opts.items():
    gdal.SetConfigOption(k, v)

# Use either /vsizip//vsis3/
path = '/vsizip//vsis3/cdn-misc.kx.gd/foss-tickets/file_example_TIFF_10MB.zip'
# or /vsizip//vsicurl/
path = '/vsizip//vsicurl/https://cdn-misc.kx.gd/foss-tickets/file_example_TIFF_10MB.zip'

gdal_tile = gdal.Open(path, gdal.GA_ReadOnly)
gdal_tile = None

You should see one or other of the following:
S3: Request at offset 4227072, after end of file
VSICURL: Request at offset 4227072, after end of file

I think in this case the program succeeds anyway in spite of trying to read beyond the end of the file, but I'm not sure if that's always the case.

Versions and provenance

macOS Sonoma 14.3.1
GDAL 3.8.5

Additional context

Wasn't too hard to make a new test file that exhibits this behaviour, so I think this must be affecting the majority of zip files, at least when the vsi-cache is on and the user tries to access the last file in the zip.

[olsen232]

Updated with steps-to-reproduce - file_example_TIFF_10MB.zip should be accessible at the URLs provided

[elpaso]

I looked at the code an this seems a false positive to me: the VSIZIP code tries to read chunk of data adding a Z_BUFSIZE of 65536 bytes, the last read request starts from (uncompressed) offset of 10147800 (which is in valid range of the uncompressed files size 10151438) but by adding the Z_BUFSIZE it lands after the file end, the cache filling code tries to honor the request asking CURL to download the data but CURL cowardly refuses to download data out of the (known) range and emits the error.

I am looking at a way to suppress this warning when possible.

[rouault]

but by adding the Z_BUFSIZE it lands after the file end,

it should probably not try to read extra bytes if the previous Read() returned less than Z_BUFSIZE bytes