Build GDAL with internal libtiff to better support COGs

[palmerj]

Can we please build GDAL with internal Libtiff support?

This will help any QGIS user adding COG raster data from cloud sources (such as AWS S3) or any GDAL scripts querying large COGs.

Background

Currently when using GDAL or projects that depend on GDAL such, reading large Cloud Optimised Geotiffs (COGs) is very slow for random access to pixel data. In some cases large COGs can have many 10s if not 100s of MBs of header data (mostly IFD array data) and If GDAL it is built with external libtiff then the whole Geotiff header needs to be downloaded before pixel data can be fetched. If GDAL is built using internal libtiff the arrays that make up most of the IFD header are not automatically downloaded - only the parts needed to specify a pixel request, which are usually a very small amount of the overall header.

Describe the solution you'd like

Change the GDAL configure script to use --with-libtiff=internal. See PR: #1088

Describe alternatives you've considered

The libtiff project could be changed to support special compilation option, but this would require changing the existing public API of the library and would be a much harder task and would require some discussion with other libtiff stakeholders to see if this is possible. Even then this would take some time to flow through to an official release.

Additional context

The source code of the internal copy of libtiff in GDAL is actually identical
to libtiff upstream. But the internal copy is compiled with a special option
to do lazy loading of the arrays to support partially downloading the COG header.

[palmerj]

See PR: #1088

[fjperini]

@palmerj Thanks for detailing everything.
I will apply this changes with other required ones.

[palmerj]

Thank you @fjperini

[fjperini]

@palmerj I will apply this change. we can close this issue
thanks for the contribution!

[Firefishy]

FYI: Debian and Ubuntu packages use external libtiff. :-(

[pabs3]

Anyone know how exactly the internal build of libtiff differs from the external build of libtiff?

[palmerj]

Older versions of official libtiff APIs didn't allow for optimised reading of the tiff header (strile offset/count values from the [Strip|Tile][Offsets/ByteCounts] array.) The TIFFReadFromUserBuffer interface was added in 4.1 so that the user can directly provide the input buffer to decompress, and hence optimise the reading of the COG header. The GDAL internal version of libtiff also has this optimisation.

[pabs3]

Hmm, but TIFFReadFromUserBuffer is part of the public libtiff API? So why does GDAL need the internal libtiff for this?

…

-- bye, pabs https://bonedaddy.net/pabs3/

[palmerj]

But it's a new API in v4.1 (I think)

[pabs3]

According to @Firefishy the external build of 4.1 doesn't work, only the internal one does.

…

-- bye, pabs https://bonedaddy.net/pabs3/