Rasterio fails when using ZSTD compression

[vincentsarago]

I've got a strange behavior when trying to create a cog with ZSTD compression resulting in python process to exit.

rio cogeo raw.tif cog_zstd.tif -p zstd
  [####################################]  100%
Killed

setting CPL_DEBUG=ON doesn't give more info.

After few investigation, it seems to be a memory error*, but I can't find the right config to make it pass.

Note:
If I remove copy_src_overviews=True in https://github.com/mapbox/rio-cogeo/blob/7c5b1893a47719cd6ea50d21c272e4b6c04b30af/rio_cogeo/cogeo.py#L100 it doesn't fail but then the file is not a COG with overview.

file used: https://github.com/mapbox/cog_cow_testsuite/blob/master/data/raw.tif

[vincentsarago]

Update:

While trying to create a cog using gdal commands I get a strange behavior, having GDAL complaining about BIGTIFF while the input or output file should be way smaller than the 4Go limit.

$ gdal_translate raw.tif ovr.tif
Input file size is 4096, 4096
0...10...20...30...40...50...60...70...80...90...100 - done.

$ gdaladdo -r average ovr.tif 2 4 8 16 32 64
0...10...20...30...40...50...60...70...80...90...100 - done.

$ gdalinfo ovr.tif
Driver: GTiff/GeoTIFF
Files: ovr.tif
Size is 4096, 4096
Coordinate System is:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs"],
    AUTHORITY["EPSG","3857"]]
Origin = (14324910.596868310123682,4535467.510329215787351)
Pixel Size = (0.149291008710861,-0.149291008710861)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (14324910.597, 4535467.510) (128d40'58.30"E, 37d41'17.40"N)
Lower Left  (14324910.597, 4534856.014) (128d40'58.30"E, 37d41' 1.75"N)
Upper Right (14325522.093, 4535467.510) (128d41'18.08"E, 37d41'17.40"N)
Lower Right (14325522.093, 4534856.014) (128d41'18.08"E, 37d41' 1.75"N)
Center      (14325216.345, 4535161.762) (128d41' 8.19"E, 37d41' 9.58"N)
Band 1 Block=4096x1 Type=Byte, ColorInterp=Red
  Overviews: 2048x2048, 1024x1024, 512x512, 256x256, 128x128, 64x64
Band 2 Block=4096x1 Type=Byte, ColorInterp=Green
  Overviews: 2048x2048, 1024x1024, 512x512, 256x256, 128x128, 64x64
Band 3 Block=4096x1 Type=Byte, ColorInterp=Blue
  Overviews: 2048x2048, 1024x1024, 512x512, 256x256, 128x128, 64x64

$ gdal_translate ovr.tif cog_zstd.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=ZSTD
Input file size is 4096, 4096
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.
.....
More than 1000 errors or warnings have been reported. No more will be reported from now.

[vincentsarago]

per OSGeo/gdal#833
we can close and wait for gdal 2.3.2 ;-)