Unexpected behavior of gdal_translate with -co COPY_SRC_OVERVIEWS=YES option for ZSTD compression

[vincentsarago]

Expected behavior and actual behavior.

This was first seen using rasterio (cogeotiff/rio-cogeo#23) but I believe it's a GDAL issue.

Steps to reproduce the problem.

$ wget 
https://raw.githubusercontent.com/mapbox/cog_cow_testsuite/master/data/raw.tif?token=AJ7PbMVo0QoCSvxgolH1psDKhdg5-6Uaks5bcy-9wA%3D%3D
$ gdal_translate raw.tif ovr.tif
$ gdaladdo -r average ovr.tif 2 4 8 16 32 64
$ gdal_translate ovr.tif cog_zstd.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=ZSTD

ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
.....
More than 1000 errors or warnings have been reported. No more will be reported from now.

The file should not be > 4Go

$ gdalinfo ovr.tif
Driver: GTiff/GeoTIFF
Files: ovr.tif
Size is 4096, 4096
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (14324910.596868310123682,4535467.510329215787351)
Pixel Size = (0.149291008710861,-0.149291008710861)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (14324910.597, 4535467.510) (128d40'58.30"E, 37d41'17.40"N)
Lower Left  (14324910.597, 4534856.014) (128d40'58.30"E, 37d41' 1.75"N)
Upper Right (14325522.093, 4535467.510) (128d41'18.08"E, 37d41'17.40"N)
Lower Right (14325522.093, 4534856.014) (128d41'18.08"E, 37d41' 1.75"N)
Center      (14325216.345, 4535161.762) (128d41' 8.19"E, 37d41' 9.58"N)
Band 1 Block=4096x1 Type=Byte, ColorInterp=Red
  Overviews: 2048x2048, 1024x1024, 512x512, 256x256, 128x128, 64x64
Band 2 Block=4096x1 Type=Byte, ColorInterp=Green
  Overviews: 2048x2048, 1024x1024, 512x512, 256x256, 128x128, 64x64
Band 3 Block=4096x1 Type=Byte, ColorInterp=Blue
  Overviews: 2048x2048, 1024x1024, 512x512, 256x256, 128x128, 64x64

Adding BIGTIFF=YES option removes the error message but the process hangs forever.

$ gdal_translate ovr.tif cog_zstd.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=ZSTD -co BIGTIFF=YES

Interesting fact, is that by removing -co COPY_SRC_OVERVIEWS=YES the command is successful.

$ gdal_translate ovr.tif cog_zstd.tif -co TILED=YES -co COMPRESS=ZSTD
Input file size is 4096, 4096
0...10...20...30...40...50...60...70...80...90...100 - done.

Operating system

Mac OS or Linux (docker)

GDAL version and provenance

2.3.1 (libzstd: 1.3.5)

cc @rouault