[GDAL] On Linux completly broken

[Neumann-A]

This port does not build correctly on linux. (even though it builds succesfully)
It does not pass any option of 3rd party libraries and the configure discovery of libraries of this port is terrible broken which can be seen by simply looking at the config-x64-linux-(rel|dbg)-out.log.

As an example: gdal depents on sqlite3 but the configure log tells:

checking for sqlite3_open in -lsqlite3... no

the reason why configure fails is due to undefined symbols due to not linking pthread and dl in the test (the script even sets LIBS='' for the test so i cannot pass those via the LIBS variable).

To solve the problem all libraries must be passed via -with-<package>=${CURRENT_INSTALLED_DIR}/(debug/) in addition to that the configure script itself must be patched. Furthermore the configure file in the debug build must be changed so that every occourence of -with-<package>/include is replaced by -with-<package>/../include due to additional header existence checks (WTF?!? Why? You already did compiler/linker checks .....)

(and that might not be all)

Here is a list of all gdal configure options for reference:

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print \`checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for \`--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or \`..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [$ac_default_prefix]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, \`make install' will install all the files in
\`$ac_default_prefix/bin', \`$ac_default_prefix/lib' etc.  You can specify
an installation prefix other than \`$ac_default_prefix' using \`--prefix',
for instance \`--prefix=\$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/PACKAGE]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]
_ACEOF

  cat <<\_ACEOF

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]
_ACEOF
fi

if test -n "$ac_init_help"; then

  cat <<\_ACEOF

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-shared[=PKGS]  build shared libraries [default=yes]
  --enable-static[=PKGS]  build static libraries [default=yes]
  --enable-fast-install[=PKGS]
                          optimize for fast installation [default=yes]
  --disable-libtool-lock  avoid locking (might break parallel builds)
  --enable-debug          enable debugging (disabled by default)
  --enable-lto            enable LTO(link time optimization) (disabled by
                          default)
  --disable-rpath         do not hardcode runtime library paths
  --enable-pdf-plugin     enable PDF driver as a plugin (included in libgdal
                          by default)

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-cpp14            Enable C++14 compiler options
  --with-pic[=PKGS]       try to use only PIC/non-PIC objects [default=use
                          both]
  --with-aix-soname=aix|svr4|both
                          shared library versioning (aka "SONAME") variant to
                          provide on AIX, [default=aix].
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-sysroot[=DIR]    Search for dependent libraries within DIR (or the
                          compiler's sysroot if not specified).
  --without-libtool     Don't use libtool to build the library
  --without-ld-shared   Disable shared library support
  --with-unix-stdio-64=ARG Utilize 64 stdio api (yes/no)
  --with-sse=ARG        Detect SSE availability for some optimized routines (ARG=yes(default), no)
  --with-ssse3=ARG        Detect SSSE3 availability for some optimized routines (ARG=yes(default), no)
  --with-avx=ARG        Detect AVX availability for some optimized routines (ARG=yes(default), no)
  --with-hide-internal-symbols=ARG Try to hide internal symbols (ARG=yes/no)
  --with-rename-internal-libtiff-symbols=ARG Prefix internal libtiff symbols with gdal_ (ARG=yes/no)
  --with-rename-internal-libgeotiff-symbols=ARG Prefix internal libgeotiff symbols with gdal_ (ARG=yes/no)
  --with-rename-internal-shapelib-symbols=ARG Prefix internal shapelib symbols with gdal_ (ARG=yes/no)
  --with-local=dir      Include /usr/local or other local tree for INCLUDE/LIBS
  --with-threads=ARG    Include thread safe support (ARG=yes(default), no or linkopt)
  --with-libz=ARG       Include libz support (ARG=internal or libz directory)
  --with-gnu-ld           assume the C compiler uses GNU ld default=no
  --with-libiconv-prefix[=DIR]  search for libiconv in DIR/include and DIR/lib
  --without-libiconv-prefix     don't search for libiconv in includedir and libdir
  --with-liblzma=ARG       Include liblzma support (ARG=yes/no)
  --with-zstd=ARG       Include zstd support (ARG=yes/no/installation_prefix)
  --with-pg=ARG           Include PostgreSQL GDAL/OGR Support (ARG=path to
                          pg_config)
  --with-grass=ARG      Include GRASS support (GRASS 5.7+, ARG=GRASS install tree dir)
  --with-libgrass=ARG   Include GRASS support based on libgrass (GRASS 5.0+)
  --with-cfitsio=ARG    Include FITS support (ARG=no or libcfitsio path)
  --with-pcraster=ARG   Include PCRaster (libcsf) support (ARG=internal, no or path)
  --with-png=ARG        Include PNG support (ARG=internal, no or path)
  --with-dds=ARG        Include DDS support (ARG=no, or path)
  --with-gta=ARG        Include GTA support (ARG=no or libgta tree prefix)
  --with-pcidsk=ARG     Path to external PCIDSK SDK or internal (default)
  --with-libtiff=ARG    Libtiff library to use (ARG=internal, yes or path)
  --with-geotiff=ARG    Libgeotiff library to use (ARG=internal, yes or path)
  --with-jpeg=ARG       Include JPEG support (ARG=internal, no or path)
  --with-charls           Include JPEG-Lossless support
  --without-jpeg12        Disable JPEG 8/12bit TIFF support
  --with-gif=ARG        Include GIF support (ARG=internal, no or path)
  --with-ogdi=ARG       Include OGDI support (ARG=path)
  --with-fme=ARG        Include FMEObjects support (ARG=FME_HOME path)
  --with-sosi=ARG        Include SOSI support (ARG=SOSI lib Path, yes or no)
  --with-mongocxx=ARG        Include MongoCXX support (ARG=Path, yes or no)
  --with-boost-lib-path=ARG   Path to boost libraries for mongocxx client
  --with-hdf4=ARG       Include HDF4 support (ARG=path)
  --with-hdf5=ARG       Include HDF5 support (ARG=path)
  --with-kea=ARG      Include kealib (ARG=path to kea-config) [default=yes]
  --with-netcdf=ARG     Include netCDF support (ARG=no or netCDF tree prefix)
  --with-jasper=ARG     Include JPEG-2000 support via JasPer library (ARG=path)
  --with-openjpeg     Include JPEG-2000 support via OpenJPEG 2.x library
  --with-fgdb=ARG        Include ESRI File Geodatabase support (ARG=FGDP API Path, yes or no)
  --with-ecw=ARG        Include ECW support (ARG=ECW SDK Path, yes or no)
  --with-kakadu=ARG     Include Kakadu/JPEG2000 support
  --with-mrsid=ARG      Include MrSID support (ARG=path to MrSID DSDK or no)
  --with-jp2mrsid=ARG   Enable MrSID JPEG2000 support (ARG=yes/no)
  --with-mrsid_lidar=ARG      Include MrSID/MG4 LiDAR support (ARG=path to LizardTech LiDAR SDK or no)
  --with-j2lura=ARG    Include JP2Lua support (ARG=no, lura SDK install path)
  --with-msg=ARG          Enable MSG driver (ARG=yes or no)
  --without-bsb           Disable BSB driver (legal issues pending
  --with-oci=[ARG]        use Oracle OCI API from given Oracle home
                          (ARG=path); use existing ORACLE_HOME (ARG=yes);
                          disable Oracle OCI support (ARG=no)
  --with-oci-include=[DIR]
                          use Oracle OCI API headers from given path
  --with-oci-lib=[DIR]    use Oracle OCI API libraries from given path
  --without-grib          Disable GRIB driver
  --with-gnm            Build GNM into shared library
  --with-mysql=ARG      Include MySQL (ARG=path to mysql_config) [default=no]
  --with-ingres=ARG     Include Ingres (ARG=$II_SYSTEM)
  --with-xerces=[ARG]     use Xerces C++ Parser from given prefix (ARG=path);
                          check standard prefixes (ARG=yes); disable (ARG=no)
  --with-xerces-inc=[DIR] path to Xerces C++ Parser headers
  --with-xerces-lib=[ARG] link options for Xerces C++ Parser libraries
  --with-expat=[ARG]      use Expat XML Parser from given prefix (ARG=path);
                          check standard prefixes (ARG=yes); disable (ARG=no)
  --with-expat-inc=[DIR]  path to Expat XML Parser headers
  --with-expat-lib=[ARG]  link options for Expat XML Parser libraries
  --with-libkml=[ARG]     use Google libkml from given prefix (ARG=path);
                          check standard prefixes (ARG=yes); disable (ARG=no)
  --with-libkml-inc=[DIR] path to Google libkml headers
  --with-libkml-lib=[ARG] link options for Google libkml libraries
  --with-odbc=ARG       Include ODBC support (ARG=no or path)
  --with-dods-root=ARG  Include DODS support (ARG=no or absolute path)
  --with-curl=ARG       Include curl (ARG=path to curl-config.)
  --with-xml2=ARG       Include libxml2 (ARG=path to xml2-config.)
  --with-spatialite=ARG Include SpatiaLite support (ARG=no(default), yes, dlopen (only supported for Spatialite >= 4.1.2) or path)
  --with-spatialite-soname=ARG Spatialite shared object name (e.g. libspatialite.so), only used if --with-spatiliate=dlopen
  --with-sqlite3=[ARG]    use SQLite 3 library [default=yes], optionally
                          specify the prefix for sqlite3 library
  --with-rasterlite2=ARG Include RasterLite2 support (ARG=no(default), yes or path)
  --with-pcre             Include libpcre support (REGEXP support for SQLite)
  --with-teigha=path Include Teigha DWG/DGN support
  --with-teigha-plt=platform Teigha platform
  --with-idb=DIR        Include Informix DataBlade support (DIR points to Informix root)
  --with-sde=DIR        Include ESRI SDE support (DIR is SDE's install dir).
  --with-sde-version=VERSION NUMBER  Set ESRI SDE version number (Default is 80).
  --with-epsilon=ARG    Include EPSILON support (ARG=no, yes or libepsilon install root path)
  --with-webp=ARG    Include WEBP support (ARG=no, yes or libwebp install root path)
  --with-geos=ARG         Include GEOS support (ARG=yes, no or geos-config
                          path)
  --with-sfcgal=ARG       Include SFCGAL support (ARG=yes, no or sfcgal-config
                          path)
  --with-qhull=ARG     Include QHull support (ARG=no, yes, internal)
  --with-opencl=ARG       Include OpenCL (GPU) support
  --with-opencl-include=ARG OpenCL Include directory (with a CL subdirectory)
  --with-opencl-lib=ARG   OpenCL Link Flags (i.e. -L/xxx -lOpenCL)
  --with-freexl=ARG    Include freexl support (ARG=no, yes (default) or libfreexl install path)
  --with-libjson-c=ARG       Include libjson-c support (ARG=internal or libjson-c directory)
  --without-pam         Disable PAM (.aux.xml) support
  --with-poppler=ARG    Include poppler(for PDF) support (ARG=no(default), yes or poppler install path)
  --with-podofo=ARG    Include podofo(for PDF) support (ARG=no(default), yes or podofo install path)
  --with-podofo-lib=ARG   podofo Link Flags (i.e. -L/xxx -lpodofo ...). Mainly for static libpodofo
  --with-podofo-extra-lib-for-test=ARG   Additional libraries to pass the detection test, but not used for libgdal linking (i.e. -ljpeg ...). Mainly for static libpodofo
  --with-pdfium=ARG    Include pdfium (for PDF) support (ARG=no(default), yes or pdfium install path)
  --with-pdfium-lib=ARG   pdfium Link Flags (i.e. -L/xxx -lpdfium ...). Mainly for static libpdfium
  --with-pdfium-extra-lib-for-test=ARG   Additional libraries to pass the detection test, but not used for libgdal linking (i.e. -ljpeg ...). Mainly for static libpdfium
  --with-proj=ARG Compile with PROJ.x (ARG=no or path)
  --with-proj5-api=ARG Allow using PROJ.5 new API  (ARG=no/yes, default to yes)
  --with-static-proj4=ARG Compile with PROJ.4 statically (deprecated, use --with-proj instead) (ARG=no or path)
  --with-gdal-ver=ARG   Override GDAL version
  --with-macosx-framework         Build and install GDAL as a Mac OS X Framework
  --with-perl           Enable perl bindings
  --with-python=ARG   Enable python bindings (ARG=yes, no, or path to python binary)
  --with-java       Include Java support (ARG=yes, no or JDK home path)  [default=no]
  --with-hdfs=ARG        Include HDFS support (ARG=Path or no)
  --with-mdb       Include MDB driver
  --with-jvm-lib=ARG        ARG is dlopen or points to Java libjvm path
  --with-jvm-lib-add-rpath    Add the libjvm path to the RPATH (no by default)
  --with-rasdaman=DIR        Include rasdaman support (DIR is rasdaman's install dir).
  --with-armadillo=ARG       Include Armadillo support for faster TPS transform computation (ARG=yes/no/path to armadillo install root) [default=no]
  --with-cryptopp=ARG       Include cryptopp support (ARG=yes, no or path)
  --with-crypto=ARG       Include crypto (from openssl) support (ARG=yes, no or path)
  --without-mrf           Disable MRF driver
  --without-lerc          Disable LERC library
  --with-null             Enable NULL driver (only useful for development
                          purposes

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags
  LT_SYS_LIBRARY_PATH
              User-defined run-time library search path.
  CPP         C preprocessor
  CXXCPP      C++ preprocessor
  PKG_CONFIG  path to pkg-config utility
  OGDI_CFLAGS C compiler flags for OGDI, overriding pkg-config
  OGDI_LIBS   linker flags for OGDI, overriding pkg-config
  HDF5_CFLAGS C compiler flags for HDF5, overriding pkg-config
  HDF5_LIBS   linker flags for HDF5, overriding pkg-config
  OPENJPEG_CFLAGS
              C compiler flags for OPENJPEG, overriding pkg-config
  OPENJPEG_LIBS
              linker flags for OPENJPEG, overriding pkg-config
  LIBKML_CFLAGS
              C compiler flags for LIBKML, overriding pkg-config
  LIBKML_LIBS linker flags for LIBKML, overriding pkg-config
  POPPLER_CFLAGS
              C compiler flags for POPPLER, overriding pkg-config
  POPPLER_LIBS
              linker flags for POPPLER, overriding pkg-config

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

Finally, I think we should break the build of the port on linux completly instead of allowing this mess.

[JackBoosY]

This is a common problem with almost every port that uses configure / make. and I meet it almost every day.
Since Linux may have installed a lot of libraries in /usr/, even if you don't pass --with-<pkg> to configure, you can build it smoothly.

I think I should add a public function like vcpkg_check_features to solve this issue. Because this configurable parameter generated using configure.ac is almost the same (--with-<pkg> and --disable-<pkg>).

What really bothers me is those ports that don't support using --with-<pkg>. We have to modify configure.ac or even configure to add this support. However, I don't know much about the shell language. These changes are difficult for me.

@ras0219-msft What do you think?

[JackBoosY]

Related: #8943 #6680.

[JackBoosY]

I added a cmake function vcpkg_use_system_ports and will test it with gdal. It has been tested using gettext and works!
I found that there is a lot of code that needs to be changed to existing functions. So I need to do a lot of work.
And building gdal is very time consuming, and the revision work will last for a long time.

[lakinwecker]

@JackBoosY I'd be happy to help with this, if it's possible. Just let me know.

[Mik-pe]

I stumbled upon a problem which I find related to this.

We tried to build proj4 with dynamic linking (since gdal 2.4.1 at least seems to want libproj.so at runtime rather than compile-time linking with it, so it seemed like a first step to build it)

I used the steps at https://github.com/microsoft/vcpkg/blob/master/docs/examples/overlay-triplets-linux-dynamic.md#overriding-default-triplets

I build proj4 through ./vcpkg install proj4:x64-linux-dynamic --overlay-triplets=vcpkg-custom-triplets with my custom triplet file looking like the guide, no problem up until this point.

However, building proj4 as a shared library seems to be non-trivial with how sqlite3 is built, the sqlite3 binary in
installed/x64-linux-dynamic/tools/sqlite3
does not find libsqlite3.so (which lies in the lib folder) and proj4 uses this binary internally through CMake.

The proj4 command that breaks for me:
proj4 data/CMakeLists.txt:49 COMMAND ${EXE_SQLITE3} -init ${ALL_SQL_IN} ${PROJ_DB} .quit break since it cannot find the .so-file.

ldd output for the sqlite3 binary:

linux-vdso.so.1 (0x00007fffc8b95000)
libsqlite3.so => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1749410000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1749c00000)

I could go and setup LD_LIBRARY_PATH to the lib folder of vcpkg, but I feel like that defeats the purpose...

[dg0yt]

I'd like to indicate that I'm trying to improve the Unix part of the gdal portfile. At the moment I'm about to establish a fit between explicit dependencies from vcpkg.json and configuration options. However, this may be a long walk, in particular with correctly handling transitive dependencies and debug variants.

[dg0yt]

It does not pass any option of 3rd party libraries and the configure discovery of libraries of this port is terrible broken

This problem extends to macOS triplets, or basically to all triplets which need configure IMO.

[dg0yt]

PRs or issues this depends on directly or transitively (unless eventually resolved by patching gdal):

• PR [vcpkg] Don't override pkg-config prefix #17205 Fix pkg-config prefix
  • PR [vcpkg baseline][libmicrohttpd] Control dependencies, doc, examples, test #17653 libmicrohttpd dependencies
• PR [geos] Relocatable geos-config; dynamic builds on Unix #17616 geos-config relocatable and available for debug builds
  • Package installation layout for tools #17607 Package installation layout for tools
• curl-config executable, relocatable and available for debug builds Instead of curl-config, use ${PKG_CONFIG} libcurl.
  • PR [curl] Fix linker flags in curl-config and libcurl.pc for osx #17790 Fix exported curl linker flags for osx
• Get correct libraries for static linking from dependencies
  • PR [tiff] Update to 4.3.0, improvements #18187 tiff requires liblzma, libjpeg (present in pc file but not used in configure test), libm, zlib (not present in pc file):

    checking for libtiff... using libtiff from /home/vsts/work/1/s/installed/x64-linux.
    checking for TIFFScanlineSize64 in -ltiff... no
    

  • PR [libxml2] Improve pc file #17747 libxml2 requires -pthread, -lm, -lzma[d], -lz, fix pc file
• PR [cfitsio] Fix pc files #17696 Fix cfitsio pc files (to be used in patched gdal)
• PR [szip, hdf5] Fix mingw import lib names, control linkage #17941 Fix szip, hdf5 pc files
• PR [netcdf-c] Fix pc files; use features #18120 Fix netcdf-c pc files
• PR [hdf5] Fix linking order in pc file #18825 Fix hdf5 linking order
• ... (to be updated)

[Neumann-A]

You might want to use:
https://github.com/microsoft/vcpkg/blob/master/ports/vcpkg-pkgconfig-get-modules/x_vcpkg_pkgconfig_get_modules.cmake

[dg0yt]

@Neumann-A Thanks. The main problem is that gdal uses (at least) three different mechanism. So my current approach is to standardize around pkg-config with patches for now, and propose similar changes to upstream. Only that I never worked with configure.ac before (because I never wanted to make autotools a requirement in my build system). For example, this is the patch for tiff:

-AC_ARG_WITH(libtiff,[  --with-libtiff=ARG    Libtiff library to use (ARG=internal, yes or path)],,)
+AC_ARG_WITH(libtiff,[  --with-libtiff=ARG    Libtiff library to use (ARG=internal, yes, pkg-config or path)],,)
+
+if test "x${with_libtiff}" = "xpkg-config" ; then
+  PKG_PROG_PKG_CONFIG([0.21])
+  PKG_CHECK_MODULES([TIFF],[libtiff-4 >= 4.0.0],[TIFF_SETTING=external],[TIFF_SETTING= HAVE_BIGTIFF=no])
+  if test "$TIFF_SETTING" = "external"; then
+    AC_MSG_RESULT([using libtiff from ${with_libtiff}.])
+    LIBS="$TIFF_LIBS $LIBS"
+    EXTRA_INCLUDES="$TIFF_CFLAGS $EXTRA_INCLUDES"
+    AC_CHECK_FUNC(TIFFScanlineSize64,HAVE_BIGTIFF=yes,HAVE_BIGTIFF=no)
+  fi
+else

[m-kuhn]

Would it help to build gdal through cmake, either by including it here or upstream?

CMakeLists exist https://github.com/miurahr/cmake4gdal (I don't know their quality)
An upstream issue exists OSGeo/gdal#3234

[dg0yt]

Would it help to build gdal through cmake, either by including it here or upstream?

CMakeLists exist https://github.com/miurahr/cmake4gdal (I don't know their quality)
An upstream issue exists OSGeo/gdal#3234

@m-kuhn AFAICT this is all too fragile, in particlar if you want to pick gdal updates quickly. For the full history, there is also OSGeo/gdal#664, and it references https://trac.osgeo.org/gdal/ticket/7080.

It is not too hard to build gdal cross-platform with autotools ... if you get the right configuration details from the dependencies.

[m-kuhn]

Thanks for the context, I wasn't aware of the trac and closed gh tickets. I'm very familiar with cross comipiling gdal. Not so much with the dependency discovery (especially in the vcpkg context but also otherwise).

[dg0yt]

This issue should be fixed now.

[m-kuhn]

Thanks for taking care @dg0yt much appreciated!

[JackBoosY]

Closed this issue via #17698 merged.
Thanks for @dg0yt hard work!