[vcpkg][gdal] Switch build method to vcpkg__nmake(win)/vcpkg__make(unix) #9100

JackBoosY · 2019-11-26T09:34:08Z

This PR will:

Update to 3.1.2
Add new option BUILD_TARGET to vcpkg_*_nmake to select target.
Add new option MAKEFILE to vcpkg_build_make to select the makefile.
Add new vcpkg function vcpkg_use_system_ports to use configure's option to handle vcpkg installed ports.
Switch gdal build method to vcpkg_*_nmake (Windows) / vcpkg_*_make (UNIX).
Fix linux build.

Related: #9068.

…nt `all` [gdal]Switch build method to vcpkg_*_nmake(win)/vcpkg_*_make(unix)

JackBoosY · 2019-11-26T09:47:52Z

@Neumann-A This PR should fix your issue. Can you review it?

Thanks.

JackBoosY · 2019-11-27T09:41:26Z

@Neumann-A You are right, I found that almost all features were broken. Mostly because the dependent libraries were not added to the test case.

Neumann-A · 2019-11-27T16:48:43Z

@JackBoosY You should probably ask @lakinwecker to check this PR.
I'm not using gdal. I was just trying to solve a problem which was discussed in the slack channel and then discovered that gdal on linux is totally broken and made an issue out of it.

ports/gdal/0004-fix-linux-configure.patch

JackBoosY · 2019-11-29T07:45:09Z

@Neumann-A In the end, I found that the order of ports in gdal configure/configure.ac is incorrect. We can only solve these configuration problems by adding -l$ {lib}.

JackBoosY · 2019-11-29T08:00:52Z

Current status:

  Installation directory:    /home/usr/work/9100/vcpkg/packages/gdal_x64-linux/debug
  C compiler:                gcc -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME  -g -DDEBUG -ftrapv
  C++ compiler:              g++ -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME  -g -DDEBUG -ftrapv
  C++14 support:             no

  LIBTOOL support:           yes

  LIBZ support:              external
  LIBLZMA support:           yes
  ZSTD support:              yes
  cryptopp support:          no
  crypto/openssl support:    yes
  GRASS support:             no
  CFITSIO support:           no
  PCRaster support:          internal
  LIBPNG support:            external
  DDS support:               no
  GTA support:               no
  LIBTIFF support:           external (BigTIFF=no)
  LIBGEOTIFF support:        internal
  LIBJPEG support:           internal
  12 bit JPEG:               no
  12 bit JPEG-in-TIFF:       no
  LIBGIF support:            external
  JPEG-Lossless/CharLS:      no
  OGDI support:              no
  HDF4 support:              no
  HDF5 support:              yes
  Kea support:               no
  NetCDF support:            yes
  NetCDF has netcdf_mem.h:   yes
  Kakadu support:            no
  JasPer support:            no
  OpenJPEG support:          yes
  ECW support:               no
  MrSID support:             no
  MrSID/MG4 Lidar support:   no
  JP2Lura support:           no
  MSG support:               no
  GRIB support:              yes
  EPSILON support:           no
  WebP support:              yes
  cURL support (wms/wcs/...):yes
  PostgreSQL support:        no
  MRF support:               yes
  LERC support:              yes
  MySQL support:             yes
  Ingres support:            no
  Xerces-C support:          yes
  NAS support:               yes
  Expat support:             yes
  libxml2 support:           yes
  Google libkml support:     no
  ODBC support:              yes
  PGeo support:              yes
  FGDB support:              no
  MDB support:               no
  PCIDSK support:            internal
  OCI support:               no
  GEORASTER support:         no
  SDE support:               no
  Rasdaman support:          no
  DODS support:              no
  SQLite support:            yes
  PCRE support:              yes
  SpatiaLite support:        yes
  RasterLite2 support:       no
  Teigha (DWG and DGNv8):    no
  INFORMIX DataBlade support:no
  GEOS support:              yes
  SFCGAL support:            no
  QHull support:             internal
  Poppler support:           no
  Podofo support:            no
  PDFium support:            no
  OpenCL support:            no
  Armadillo support:         no
  FreeXL support:            no
  SOSI support:              no
  MongoDB support:           no
  HDFS support:              no
  userfaultfd support:       yes


  SWIG Bindings:             no

  Statically link PROJ:      yes
  enable GNM building:       yes
  enable pthread support:    yes
  enable POSIX iconv support:yes
  hide internal symbols:     no

JackBoosY · 2019-11-29T08:31:45Z

The CI system did not really test gdal, we need to fix it before merging this PR.

Neumann-A · 2019-11-29T08:38:28Z

  LIBGEOTIFF support:        internal
  LIBJPEG support:           internal
  QHull support:             internal

shouldn't those be external? (If I understand external correctly)
It also looks like some of those dependencies could potentially screw up CI results (if there is a port depending on gdal)
like

  Armadillo support:         no
  MongoDB support:           no
  PostgreSQL support:        no

JackBoosY · 2019-11-29T08:47:33Z

@Neumann-A Because these 6 ports are not added as functions. This PR only fixes existing features.

vicroms · 2019-12-02T23:24:18Z

liblzma and libjpeg-turbo are failing for Linux, that causes tiff and gdal to be skipped by CI.

grdowns · 2019-12-02T23:32:47Z

Hey @JackBoosY. We like vcpkg_use_system_port, but @vicroms and I are thinking about whether it can be merged into vcpkg_check_features with a flag. We'll take this on and ping you when we've got something for you to see!

ras0219-msft · 2019-12-02T23:47:46Z

Explanation of common autoconf-style arguments: https://autotools.io/autoconf/arguments.html

It looks like "enable" is typically paired with "disable" and "with" is paired with "without".

JackBoosY · 2019-12-03T05:42:31Z

@ras0219-msft According to the documentation, with should be paired with without, but in general, the judgment in configure.ac is:

elif test "$with_libgrass" = "no" ; then

or

if test "$with_pcidsk" != "no" ; then

janholt · 2019-12-03T19:17:39Z

@JackBoosY gdal[core] (commit cb7d59e Add feature options) works for me on x64-windows and x64-linux with small changes:

diff --git a/ports/gdal/portfile.cmake b/ports/gdal/portfile.cmake
index 34af0075b..1b938ffd7 100644
--- a/ports/gdal/portfile.cmake
+++ b/ports/gdal/portfile.cmake
@@ -119,6 +119,7 @@ if (VCPKG_TARGET_IS_WINDOWS)
         HTMLDIR=${NATIVE_HTML_DIR}
         GEOS_DIR=${COMMON_INCLUDE_DIR}
         "GEOS_CFLAGS=-I${COMMON_INCLUDE_DIR} -DHAVE_GEOS"
+        "PROJ_FLAGS=-DPROJ_STATIC -DPROJ_VERSION=5"
         PROJ_INCLUDE=-I${COMMON_INCLUDE_DIR}
         EXPAT_DIR=${COMMON_INCLUDE_DIR}
         EXPAT_INCLUDE=-I${COMMON_INCLUDE_DIR}
@@ -188,7 +189,7 @@ if (VCPKG_TARGET_IS_WINDOWS)
         OPTIONS_RELEASE ${NMAKE_OPTIONS_REL}
     )
     
-    if (CMAKE_BUILD_TYPE STREQUAL debug)
+    if (VCPKG_BUILD_TYPE STREQUAL debug)
         set(GDAL_OBJ_DIR ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-dbg)
     else()
         set(GDAL_OBJ_DIR ${CURRENT_BUILDTREES_DIR}/${TARGET_TRIPLET}-rel)
@@ -281,7 +282,12 @@ else()
         SOURCE_PATH ${SOURCE_PATH}
         AUTOCONFIG
         OPTIONS ${EXTRA_OPTIONS}
+            --with-jpeg=${CURRENT_INSTALLED_DIR}
+            --with-qhull=no
             --with-proj5-api=yes
+            --with-hide-internal-symbols=yes
+            --with-rename-internal-libgeotiff-symbols=yes
+            --with-rename-internal-shapelib-symbols=yes
             --enable-shared=${BUILD_DYNAMIC}
             --enable-static=${BUILD_STATIC}
         OPTIONS_DEBUG
@@ -303,11 +309,14 @@ else()
     
     file(GLOB GDAL_TOOLS ${CURRENT_PACKAGES_DIR}/bin/*)
     foreach (GDAL_TOOL ${GDAL_TOOLS})
-        file(INSTALL ${GDAL_TOOL} DESTINATION ${CURRENT_PACKAGES_DIR}/tools/gdal)
+        file(INSTALL ${GDAL_TOOL} DESTINATION ${CURRENT_PACKAGES_DIR}/tools/gdal
+                PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_WRITE GROUP_EXECUTE WORLD_READ)
         file(REMOVE ${GDAL_TOOL})
     endforeach()
     file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/bin)
 endif()
 
+configure_file(${CMAKE_CURRENT_LIST_DIR}/vcpkg-cmake-wrapper.cmake ${CURRENT_PACKAGES_DIR}/share/${PORT}/vcpkg-cmake-wrapper.cmake @ONLY)
+
 # Handle copyright

and help in vcpkg-cmake-wrapper.cmake for static linking:

_find_package(${ARGS})

if(TARGET GDAL::GDAL AND "@VCPKG_LIBRARY_LINKAGE@" STREQUAL "static")
	find_package(PROJ4 REQUIRED)
	find_package(PNG REQUIRED)
	find_library(GDAL_GEOS_LIB NAMES geos)
	find_library(GDAL_GEOSC_LIB NAMES geos_c)
	message(STATUS "GDAL_GEOS: ${GDAL_GEOS_LIB} ${GDAL_GEOSC_LIB}")
	find_package(WebP REQUIRED)
	find_package(sqlite3 CONFIG REQUIRED)
	find_package(EXPAT REQUIRED)
	find_package(OpenJPEG CONFIG REQUIRED)
	find_package(netCDF REQUIRED)
	find_package(HDF5 CONFIG REQUIRED)
	find_package(SZIP REQUIRED)
	find_package(JPEG REQUIRED)
	find_package(TIFF REQUIRED)
	find_package(Threads)
	find_package(CURL CONFIG REQUIRED)
	find_package(OpenSSL REQUIRED)
	find_package(LibXml2 REQUIRED)
	find_package(ZLIB REQUIRED)
	find_package(LibLZMA CONFIG REQUIRED)

	set_property(TARGET GDAL::GDAL APPEND PROPERTY INTERFACE_LINK_LIBRARIES
		PROJ4::proj
		PNG::PNG
		${GDAL_GEOS_LIB} ${GDAL_GEOSC_LIB}
		WebP::webp
		sqlite3
		EXPAT::EXPAT
		openjp2
		netcdf
		hdf5::hdf5_hl-static hdf5::hdf5-static 
		szip-static
		JPEG::JPEG
		TIFF::TIFF
		Threads::Threads
		${CMAKE_DL_LIBS}
		CURL::libcurl
		OpenSSL::SSL OpenSSL::Crypto
		LibXml2::LibXml2
		ZLIB::ZLIB
		LibLZMA::LibLZMA
	)
endif()

Here additional features omitted, libgeotiff - internal renamed, jpeg - external, qhull - disabled (all just for quick test).
Also I have a pretty clean Linux, with minimum -dev packages. I think it is a problem that configure can pick up "system" libraries, so maybe it is preferable to have exhaustive options list for configure that mimics configure --help (all would be explicit and easy to check that nothing is missing). For example gdal has -with-kea=ARG Include kealib (ARG=path to kea-config) [default=yes], so now you can easily end up with misterious kea.

janholt · 2019-12-07T15:05:09Z

@JackBoosY small typo fix

diff --git a/ports/gdal/portfile.cmake b/ports/gdal/portfile.cmake
index 1b938ffd7..c101cd235 100644
--- a/ports/gdal/portfile.cmake
+++ b/ports/gdal/portfile.cmake
@@ -27,8 +27,8 @@ vcpkg_extract_source_archive_ex(
 )
 
 if (VCPKG_TARGET_IS_WINDOWS)
-    set(NATIVE_PACKAGES_DIR_DBG "${CURRENT_PACKAGES_DIR}")
-    set(NATIVE_PACKAGES_DIR_REL "${CURRENT_PACKAGES_DIR}/debug")
+    set(NATIVE_PACKAGES_DIR_REL "${CURRENT_PACKAGES_DIR}")
+    set(NATIVE_PACKAGES_DIR_DBG "${CURRENT_PACKAGES_DIR}/debug")
     # We can't pass in the normal absolute path because the "xcopy" command needs to use the windows path.
     # We also can't pass in the windows path because it is used as an argument after the "\" will be used as an escape character and cannot be resolved using double quotes.
     # Use relative path here to install manually below

…jack/9068

JackBoosY · 2020-01-17T09:50:45Z

/azp run

Neumann-A · 2020-04-07T07:10:31Z

ports/gdal/0004-fix-linux-build.patch

+
+-if test "$with_liblzma" = "yes" ; then
+if test "$with_liblzma" != "no" ; then
+   AC_CHECK_LIB(lzma,lzma_code,LIBLZMA_SETTING=yes,LIBLZMA_SETTING=no,)


Don't use AC_CHECK_LIB due to possible debug suffixes. Instead use AC_SEARCH_LIBS

There are too many AC_CHECK_LIB in the source, so I think we should keep it as it is.

I mean you can also set the autotool variable by hand and pass it to configure if you feel like it but it general all those check libs calls need to be replaced with search lib calls.

@Neumann-A As my knowledge of configure is limited, can you help me do this?

Neumann-A · 2020-04-07T07:13:38Z

ports/gdal/0004-fix-linux-build.patch

+
+-  if test "$LIBLZMA_SETTING" = "yes" -a "$ac_cv_header_lzma_h" = "yes" ; then
+  if test "$LIBLZMA_SETTING" = "yes" ; then
+     LIBS="-llzma $LIBS"


Don't link by name. Use $ac_cv_search_(function) (in this case $ac_cv_search_lzma_code). Requires a check if $ac_cv_search_(function) is not equal to none required. Compare with the libwandio changes in 2e711ae

Neumann-A · 2020-04-07T07:14:06Z

ports/gdal/0004-fix-linux-build.patch

+     AC_CHECK_LIB(hdf5,H5Fopen,HAVE_HDF5=yes,HAVE_HDF5=no,)
+
+     if test "$HAVE_HDF5" = "yes" ; then
+-        LIBS="-lhdf5 $LIBS"
+        LIBS="-lhdf5_hl -lhdf5 -lszip $LIBS"


same as above
I won't comment on all the AC_CHECK_LIB and named links below but those should be changed as described above.

Neumann-A · 2020-04-07T07:15:27Z

ports/gdal/0004-fix-linux-build.patch

+   if test "$TIFF_SETTING" = "external" ; then
+-    LIBS="-ltiff $LIBS"
+    LIBS="-ltiff -llzma $LIBS"
+     AC_MSG_RESULT([using pre-installed libtiff.])
+   else
+     AC_MSG_RESULT([using internal TIFF code.])
+@@ -1667,10 +1666,10 @@ else
+
+   TIFF_SETTING=external
+   if test -r "$with_libtiff/tiff.h" ; then
+-    LIBS="-L$with_libtiff -ltiff $LIBS"
+    LIBS="-L$with_libtiff -ltiff -llzma $LIBS"
+     EXTRA_INCLUDES="-I$with_libtiff $EXTRA_INCLUDES"
+   else
+-    LIBS="-L$with_libtiff/lib -ltiff $LIBS"
+    LIBS="-L$with_libtiff/lib -ltiff -llzma $LIBS"
+     EXTRA_INCLUDES="-I$with_libtiff/include $EXTRA_INCLUDES"
+   fi


There should be a search for tiff, Why is lzma linked in again? It should already be a part of LIBS

Neumann-A · 2020-04-07T07:16:47Z

ports/gdal/0004-fix-linux-build.patch

+ elif test "$with_netcdf" = "yes" -o "$with_netcdf" = "" ; then
+
+-  AC_CHECK_LIB([netcdf], [nc_open], [NETCDF_SETTING=yes], [NETCDF_SETTING=no],)
+  AC_CHECK_LIB([netcdf], [nc_open], [NETCDF_SETTING=yes], [NETCDF_SETTING=no], -lcurl -lssl -lcrypto)


curl has a debug suffix so this won't work in debug builds. Better do a search for CURL.

ports/gdal/0004-fix-linux-build.patch

scripts/cmake/vcpkg_build_nmake.cmake

…ack/9068

JackBoosY · 2020-08-19T07:22:18Z

This is too hard for me, close it temporary.

[vcpkg_*_nmake]Add new option DISABLE_ALL to disable add build argume…

e09f639

…nt `all` [gdal]Switch build method to vcpkg_*_nmake(win)/vcpkg_*_make(unix)

JackBoosY added the info:internal This PR or Issue was filed by the vcpkg team. label Nov 26, 2019

JackBoosY requested review from vicroms and PhoebeHui November 26, 2019 09:34

Add new vcpkg function vcpkg_use_system_ports

e0334bd

grdowns self-assigned this Nov 26, 2019

JackBoosY added 2 commits November 26, 2019 23:35

[gdal]Fix install in Windows and fix configure in Linux.

45e82f0

[gdal]Add some features in Linux, enable debug

d71759a

Neumann-A reviewed Nov 27, 2019

View reviewed changes

ports/gdal/0004-fix-linux-configure.patch Outdated Show resolved Hide resolved

Neumann-A reviewed Nov 27, 2019

View reviewed changes

ports/gdal/0004-fix-linux-configure.patch Outdated Show resolved Hide resolved

[gdal]Fix linux build, add features.

3392dc7

[gdal]Recursively delete the folder bin.

1df6281

JackBoosY marked this pull request as ready for review November 29, 2019 08:27

Add feature options.

cb7d59e

JackBoosY mentioned this pull request Dec 2, 2019

How do I link GDAL to library compile #8664

Closed

grdowns assigned vicroms Dec 2, 2019

JackBoosY added 2 commits January 13, 2020 18:29

Merge branch 'master' of https://github.com/Microsoft/vcpkg into dev/…

797babc

…jack/9068

[gdal] Fix link issue on OSX

47c19f9

longhuan2018 mentioned this pull request Apr 1, 2020

[gdal] Update to 2.4.4 #10563

Closed

PhoebeHui approved these changes Apr 7, 2020

View reviewed changes

PhoebeHui added the needs-feature-review label Apr 7, 2020

PhoebeHui changed the title ~~[gdal] Switch build method to vcpkg_*_nmake(win)/vcpkg_*_make(unix)~~ [vcpkg][gdal] Switch build method to vcpkg_*_nmake(win)/vcpkg_*_make(unix) Apr 7, 2020

Neumann-A reviewed Apr 7, 2020

View reviewed changes

ports/gdal/0004-fix-linux-build.patch Show resolved Hide resolved

Neumann-A reviewed Apr 7, 2020

View reviewed changes

scripts/cmake/vcpkg_build_nmake.cmake Outdated Show resolved Hide resolved

strega-nil added requires:discussion and removed legacy:needs-feature-review labels May 21, 2020

PhoebeHui marked this pull request as draft July 14, 2020 02:52

JackBoosY mentioned this pull request Jul 30, 2020

GDAL update to 3.1.0 #11905

Closed

JackBoosY added category:port-bug The issue is with a library, which is something the port should already support category:port-update The issue is with a library, which is requesting update new revision labels Jul 31, 2020

JackBoosY added 4 commits July 31, 2020 02:00

erge branch 'master' of https://github.com/microsoft/vcpkg into dev/j…

1050039

…ack/9068

Replace option DISABLE_ALL to BUILD_TARGET

d324f25

Add option MAKEFILE

bf9367f

[gdal] Update to 3.1.2

b07d039

Gigas002 mentioned this pull request Aug 7, 2020

Native dependencies bugs MaxRev-Dev/gdal.netcore#23

Closed

JackBoosY closed this Aug 19, 2020

JonLiu1993 mentioned this pull request Aug 26, 2020

[gdal] update to <3.1.2> #13149

Closed

JackBoosY mentioned this pull request Sep 10, 2020

[gdal] Update to 3.1.3 #13449

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg][gdal] Switch build method to vcpkg__nmake(win)/vcpkg__make(unix) #9100

[vcpkg][gdal] Switch build method to vcpkg__nmake(win)/vcpkg__make(unix) #9100

JackBoosY commented Nov 26, 2019 •

edited

Loading

JackBoosY commented Nov 26, 2019

JackBoosY commented Nov 27, 2019

Neumann-A commented Nov 27, 2019

JackBoosY commented Nov 29, 2019

JackBoosY commented Nov 29, 2019

JackBoosY commented Nov 29, 2019

Neumann-A commented Nov 29, 2019

JackBoosY commented Nov 29, 2019

vicroms commented Dec 2, 2019

grdowns commented Dec 2, 2019

ras0219-msft commented Dec 2, 2019

JackBoosY commented Dec 3, 2019

janholt commented Dec 3, 2019

janholt commented Dec 7, 2019

JackBoosY commented Jan 17, 2020

Neumann-A Apr 7, 2020

JackBoosY Aug 3, 2020

Neumann-A Aug 3, 2020

JackBoosY Aug 3, 2020

Neumann-A Apr 7, 2020

Neumann-A Apr 7, 2020 •

edited

Loading

Neumann-A Apr 7, 2020

Neumann-A Apr 7, 2020

JackBoosY commented Aug 19, 2020

[vcpkg][gdal] Switch build method to vcpkg_*_nmake(win)/vcpkg_*_make(unix) #9100

[vcpkg][gdal] Switch build method to vcpkg_*_nmake(win)/vcpkg_*_make(unix) #9100

Conversation

JackBoosY commented Nov 26, 2019 • edited Loading

JackBoosY commented Nov 26, 2019

JackBoosY commented Nov 27, 2019

Neumann-A commented Nov 27, 2019

JackBoosY commented Nov 29, 2019

JackBoosY commented Nov 29, 2019

JackBoosY commented Nov 29, 2019

Neumann-A commented Nov 29, 2019

JackBoosY commented Nov 29, 2019

vicroms commented Dec 2, 2019

grdowns commented Dec 2, 2019

ras0219-msft commented Dec 2, 2019

JackBoosY commented Dec 3, 2019

janholt commented Dec 3, 2019

janholt commented Dec 7, 2019

JackBoosY commented Jan 17, 2020

Neumann-A Apr 7, 2020

Choose a reason for hiding this comment

JackBoosY Aug 3, 2020

Choose a reason for hiding this comment

Neumann-A Aug 3, 2020

Choose a reason for hiding this comment

JackBoosY Aug 3, 2020

Choose a reason for hiding this comment

Neumann-A Apr 7, 2020

Choose a reason for hiding this comment

Neumann-A Apr 7, 2020 • edited Loading

Choose a reason for hiding this comment

Neumann-A Apr 7, 2020

Choose a reason for hiding this comment

Neumann-A Apr 7, 2020

Choose a reason for hiding this comment

JackBoosY commented Aug 19, 2020

[vcpkg][gdal] Switch build method to vcpkg__nmake(win)/vcpkg__make(unix) #9100

[vcpkg][gdal] Switch build method to vcpkg__nmake(win)/vcpkg__make(unix) #9100

JackBoosY commented Nov 26, 2019 •

edited

Loading

Neumann-A Apr 7, 2020 •

edited

Loading