Memory leak with 3.8.x and gdal_translate from a large VRT

[trharris78]

I have a task that joins a bunch of GeoTIFF tiles with a VRT, then translates the VRT to one big GeoTIFF. When I ran this task in AWS in a Docker container, the task failed due to memory usage.

I originally observed this behavior with GDAL 3.8.1. I tried it both in the official Docker image, ghcr.io/osgeo/gdal:ubuntu-small-3.8.1, and on OSX natively (not in Docker). I also tried the official 3.8.0 Docker image and it appears to happen there, but not with the official 3.7.3 image. So possibly some change introduced in 3.8.0 is causing this.

My original data was 346 separate TIFs, approximately 3 GB, wrapped by a VRT. I have since tried to come up with a reasonable reproduce case, but it requires a large amount of input files for the ever-increasing memory usage to become obvious. I have attached one TIF, template.tif, and a simple Python script, create_test_data.py. The script will duplicate the TIF and edit each copy's geotransform in order to make a spatially extended grid, then it will make a VRT from all the tiles in the grid. The script is set up to make a grid of 20x20 tiles and takes 1-2 minutes to run on my 2019 MacBook. The simulated grid takes up about 460MB on disk.

I am using memory-profiler to produce the plots below. The gdal_translate command runs within mprof run to collect a memory trace. The plot can be displayed with mprof plot.

Steps to reproduce:

1. python create_test_data.py
2. mprof run -o mprof.dat gdal_translate -co BIGTIFF=YES -co TILED=YES -co COMPRESS=DEFLATE -co NUM_THREADS=ALL_CPUS --config GDAL_TIFF_INTERNAL_MASK YES test_data.vrt test_data.tif
3. mprof plot mprof.dat

This first plot is from a run in the official 3.8.1 Docker image. I started the Docker container with -m 8000m to tell Docker to provide only 8GB of RAM to the container. The gdal_translate command ran for a bit but was killed once memory usage reached 8GB.

Command output:

Input file size is 97920, 97920
0...10...20...30...40...50...60.root@42186bc7788c:/data#

Memory usage:

I thought some of the caching configuration options might help, so I set these very small values:

export GDAL_CACHEMAX=10 VSI_CACHE=FALSE VSI_CACHE_SIZE=1000000 GDAL_MAX_DATASET_POOL_SIZE=2 GDAL_MAX_DATASET_POOL_RAM_USAGE=10MB

I ran the same command, and this time it did complete, but the memory usage still increased continuously. It just didn't hit the 8GB limit I set for the Docker container:

Running the exact same command in the older Docker image, ghcr.io/osgeo/gdal:ubuntu-small-3.7.3, produces this much more reasonable memory profile:

Apologies for the zip file, GitHub wouldn't let me upload the .tif and .py separately.

attachments.zip

Edit: I should note that I did see the open issue #7908 but it didn't seem to have any applicable resolution.

[rouault]

Thanks for the high quality report (here and in 8968) ! Fixed queued in #8969 (for 3.9) and #8970 (for 3.8.2)