gdaladdo slowness for large files

[Wardrop]

Just carrying over this ticket from the old trac-based bug tracker: https://trac.osgeo.org/gdal/ticket/5067

Expected behavior and actual behavior.

gdaladdo becomes excruciatingly slow when working with larger datasets. Original ticket points at the dimensions being the main contributing metric, rather than file size. In my case, a 130,000 x 170,000 GeoTIFF was processed in under 12 hours with constant single-core CPU usage. On the other hand, a 519,398 x 915,156 GeoTIFF (same compression; roughly 4x larger file) went for 3 days and barely had made any progress at all. 5% progress at the most. Most of the time the CPU was sitting idle, but disk usage was constant.

Steps to reproduce the problem.

The following gdaladdo command was used:

gdaladdo --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 85 --config PHOTOMETRIC_OVERVIEW YCBCR --config BIGTIFF_OVERVIEW YES "D:\2015_25cm_mosaic_jpeg85.tif" 2 4 8 16 32 64

Adding '-ro' to the above to force generation of an external overview increases the speed by at least one order of magnitude. The original ticket indicates that this is specific to internal overviews.

Operating system

Windows 10 64-bit

GDAL version and provenance

GDAL 2.4.1

[palmerj]

This is a workaround for this issue that make things much faster (as explain by Even in https://trac.osgeo.org/gdal/ticket/5067): https://gist.github.com/palmerj/2e4a46fbcf0c97212e6e77fced22e885 This script has been designed to create large Cloud Optimised Geotiffs with a mask

[matsamentet]

Adding an enthusiastic thumbs up for an optimization of GdalAddo. I create a lot of large raster GeoPackages, where external overviews are not possible, and so the workaround is not an option.