GDAL3.0 performances issue!

[vincentsarago]

Starting in #10, we found a huge performance downgrade when doing reprojection (Warp) of dataset.
As pointed in #10 (comment) and #10 (comment) this only happens in the containers we are building (the performance in gdal official images are fine).

To Do

• Check PROJ compilation
• Check GDAL compilation

[vincentsarago]

Command

CPL_TIMESTAMP=ONE CPL_DEBUG=ON GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR gdalwarp -q -te -11036283.891926888  10410111.756214723 -10958012.374962866  10331840.239250703 -t_srs EPSG:3857 -ts 256 256 -r bilinear -overwrite /vsicurl/https://s3.amazonaws.com/opendata.remotepixel.ca/bench_tiler/LC08_L1TP_040013_20191014_20191029_01_T1_B4.tif out.tif

Numbers

GDAL 3.0
17:26:43.314265	264ms	HTTP: libcurl/7.61.1 OpenSSL/1.0.2k zlib/1.2.8 libidn2/0.16 libpsl/0.6.2 (+libicu/50.1.2) libssh2/1.4.2 nghttp2/1.21.1
17:26:43.662722	671ms	GDAL: Computing area of interest: -103.586, 65.7283, -97.4556, 68.0874
17:26:44.460977	625ms	GTiff: Opened 134x135 overview.
17:26:45.088925	588ms	GDAL: Computing area of interest: -103.586, 65.7283, -97.4556, 68.0874
17:26:45.676688	189ms	VSICURL: Downloading 2145148-2175608 (https://s3.amazonaws.com/opendata.remotepixel.ca/bench_tiler/LC08_L1TP_040013_20191014_20191029_01_T1_B4.tif)...
17:26:45.870161	124ms	GDAL: GDALWarpKernel()::GWKRealCase() Src=1544,201,252x252 Dst=0,0,256x256


GDAL 3.0 Alpine
17:42:55.857475	284ms	HTTP: libcurl/7.65.1 OpenSSL/1.1.1c zlib/1.2.11 nghttp2/1.38.0
17:42:56.273597	43ms     GDAL: Computing area of interest: -103.586, 65.7283, -97.4556, 68.0874
17:42:56.421772	15ms	    GTiff: Opened 134x135 overview.
17:42:56.440615	16ms     GDAL: Computing area of interest: -103.586, 65.7283, -97.4556, 68.0874
17:42:56.456941	210ms	VSICURL: Downloading 2145148-2175608 (https://s3.amazonaws.com/opendata.remotepixel.ca/bench_tiler/LC08_L1TP_040013_20191014_20191029_01_T1_B4.tif)...
17:42:56.725692	44ms	    GDAL: GDALWarpKernel()::GWKRealCase() Src=1544,201,252x252 Dst=0,0,256x256

Note: The timing is calculated between time of line and time of the next line, so for example

timestamp	delta	log
17:26:43.652666	10ms	GDAL: Using GTiff driver
17:26:43.662722	671ms	GDAL: Computing area of interest: -103.586, 65.7283, -97.4556, 68.0874
17:26:44.333315	1ms	GDAL: QuietDelete(out.tif) invoking Delete()

The time difference is then located somewhere here https://github.com/OSGeo/gdal/blob/release/3.0/gdal/apps/gdalwarp_lib.cpp#L939-L1040
(bewteen GDALComputeAreaOfInterest and GDALWarpCreateOutput)

full logs: https://gist.github.com/vincentsarago/2c55d5154d352f10e85d09fe9008e365

Note: GDAL 2.4 and GDAL3.0 Alpine give almost the same result, so I don't think we can say the libraries version (curl, nghttp2 ...) can be the cause.

[vincentsarago]

ref OSGeo/PROJ#1721

Basically we discovered that while PROJ 6 compiled and worked against CentOS 7
system sqlite, the query optimized of this sqlite version did a bad job that
could result in dramatic performance issues as you found. So the fix is to run
against a not too old sqlite. We have set the minimum to sqlite 3.11.