Native dependencies bugs

[Gigas002]

Problem

While using prebuilt gdal from gisinternals (and nuget packages by @szekerest), I've noticed some issues, which I've explained in details in gdal repo:

• Using wildcard breaks gdalbuildvrt osgeo/gdal - issue1749
• Bad support for cyrillic paths osgeo/gdal - issue 1245

These issues also appears in this library. Recently I've located a new problem, related to this nuget package: my antivirus deletes the package instantly because of libgeos_c.dll.

Solution

Not so long ago I've found some time to sort out how gdal builds and managed to find a solution for these issues.

First two I are fixed by changing gdal's nmake.opt. It requires a new dependency -- iconv (I've got it through vcpkg) and adding those paths to nmake.opt:

LIBICONV_INCLUDE = -IC:\vcpkg\installed\x64-windows\include
LIBICONV_LIBRARY = C:\vcpkg\installed\x64-windows\lib\libiconv.lib
LIBICONV_CFLAGS = -DICONV_CONST=const

I've also specified wsetargv.obj in nmake.opt, so it can be a part of solution too:

SETARGV = "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\lib\x64\wsetargv.obj"

I've tested the fixes above on latest gdal 3.1.2 with PROJ 7.1.0 (installed it's deps through vcpkg, built with cmake+vs using sqlite3 and libtiff) built on win-x64. Even though I've tested it on gdal apps, not the library, I think that might be also a solution for resolving the same issue in library.

Regarding the geos issue, I've tried to build geos with your win/getgeos.bat script, but antivirus still deleted the libgeos_c.dll. Building the newer commit (e.g. 3.8.1 release, not 3.8.0) fixes the issue. I suppose there were some kind of vulnerability in 3.8.0, idk.

I'm not sure how your binaries are built and don't want to break anything, so I've decided to open an issue instead of PR to convey this information to you.

[Gigas002]

I would also like to add a new dependency-related problem:
SpatialReference.ExportToPROJJSON(out string) method throws an exception, saying that it requires PROJ 6.2 or higher. I've tried building PROJ 7.1 and it seems the same as current version, so I propose to use the latest version.

[MaxRev-Dev]

@Gigas002 Thanks for this investigation.

This build system is very simple and not stable (there are no CI build tests).

• fetching sources (on windows this is done by one of get{component}.bat files.
• building them - is the second part of batch files.

Next steps, in my opinion, is to union configuration variables from GdalCore.opt and CONFIGVARS.vat and then rewrite windows part to nmake/cmake.

Issue review:

1. Good find - wildcards and cyrillic paths.
2. About antivirus - can't repro that, win10 standard windows defender with latest update did not find any threats.
3. We can export SETARGV variable to CONFIGVARS.bat, but it should be checked for path before build or set a relative path to vcvarsall64.bat.

There are several ways we can add libiconv library to build pipeline.

• create a batch script and fetch sources from git and then compile them (pros: original sources, cons: maybe 7z needed).
• provide a compiled sources (there can be some license compliance issues, but we already use swig, tcl, vc++ binaries with no license review).
• add vcpkg - this is a new dependency and maybe only for this library. It does not provide all libraries we need and also the fresh versions of other libraries like PROJ,GEOS, etc are not available.

Binaries that we can use (with 7z as another dep):
https://github.com/pffang/libiconv-for-Windows/tree/v1.16

[Gigas002]

Last time, when I've build Gdal (ver. 3.1.2), I've used the vcpkg. As you know, the PROJ and Gdal itself out there is pretty outdated. So I've pulled the PROJ dependencies (minimal, vcpkg install sqlite3:x64-windows tiff:x64-windows) and then built PROJ 7.1.0 from source (full guide), using this cmake and VS2019 (16.6.5 to be more precise).
I've took libiconv for gdal from vcpkg too (vcpkg install libiconv:x64-windows). The Gdal seems to built and work fine after this.

I also think, that gdal's port on vcpkg will be updated in the upcoming release, look at this issue and a linked PR (they're also switching the build methods in this PR, so that might be interesting/useful). Don't know much details, but probably if they'll be able to build 3.1.x Gdal that means, that they're updating all dependencies for it too? Anyway, we can request ports adding/updating on vcpkg repo (I've done it for PROJ 7.1.0 and someone's already grabbed the thing).

So I suppose that would be a good idea to pull all the needed dependencies by vcpkg, since it's also very easy to build and use (and easier to automate, I suppose), but we might need to wait until the ports will be added/updated. For now we could list all the needed dependencies and check their versions/availability on vcpkg.

Regarding the antivirus issue, I was using Kaspersky antivirus, when it deleted the libgeos_c.dll. I've tried to build geos from source with cmake/vs, but lib from 3.8.0 build is still deleted by antivirus (so it's not current repo's problem obviously). Guess we can safely update it to the latest stable release, since it doesn't break anything. (BTW, it seems, like geos will be updated in vcpkg soon too)

[MaxRev-Dev]

Several months ago, I had some experience with vcpkg.
Of course, we can use vcpkg as package manager for pulling outdated libraries.
But what about new releases in vcpkg - that not worth it . Your issue for PROJ 7.1.0 is open for 24 days with no updates. The same is for GDAL 3.1 - they are moving, but that thread is too buggy on unix.

And more one thing, I'm using concrete versions of main libraries. With vcpkg we can't specify them directly in builds.

gdal.netcore/win/CONFIGVARS.bat

Lines 5 to 12 in 70ab3da

	set GDAL_REPO=https://github.com/OSGeo/gdal.git
	set GDAL_COMMIT_VER=a3f8b8b84d38c65717e1d9f08f8be85782cf7094
	set PROJ6_REPO=https://github.com/OSGeo/PROJ.git
	set PROJ6_COMMIT_VER=2228592582910d866b8c1b5187bc2ef50955b281
	set GEOS_REPO=https://github.com/libgeos/geos.git
	set GEOS_COMMIT_VER=ff05d9755d189771147acb3105bd9c9cfff730ff

I'm not against using vpckg. Furthermore, integration will bring us more drivers and a bit of automation for windows build pipeline.

Let's establish a list of packages that we can pull using vcpkg:

• curl
• sqlite3
• tiff
• geotiff
• jpeg
• png
• iconv
• zlib
• hdf5

Other libraries optional:

• hdf4 [missing]
• hdfs [available, new for us]
• mongo-cxx-driver[boost] [new]
• netcdf-c [new]
• xerces [new]
• expat [new]
• kml [new]

[Gigas002]

Good point about versioning. Judging by vcpkg's roadmap, they're planning to support specifiying versions explicitly (yet it was planned for june release and still not released). It would be great if they'll complete this roadmap since it contains nice features. Gdal itself doesn't release that frequently, so I guess using slightly outdated packages between releases is fine.
I also browsed their release history, and it seems like they're trying to release a new version each month, but this year has gone wrong. Even if it's not, well, very good for now, I think that it'll be a great thing in the future.
I'll take a look at the packages latest versions vs vcpkg available versions and add an issues to their repository. At least with that I can help.

[MaxRev-Dev]

@Gigas002 The work is in progress.
Finally, I rewrote a windows part to nmake (almost all batch files).
I did compiled GDAL for both runtimes, but have not yet adjusted docs and makefiles for gathering binaries.

About VCPKG integration with GDAL:
That was a big problem to configure right both PROJ and GDAL - I've spent a half of a week for this.

Just a moment with configuring PROJ was insane:

gdal.netcore/unix/gdal-makefile

Lines 56 to 70 in bff4708

	configure_proj:
	@cd $(PROJ_ROOT) && chmod +x ./autogen.sh
	cd $(PROJ_ROOT) && ./autogen.sh
	# 1. --with-curl=[path] does not work because curl-config not found
	# 2. vcpkg's tiff_x64-linux default location in pkg-config also not found
	cd $(PROJ_ROOT) && ./configure --prefix=$(BUILD_ROOT)/proj-build \
	LIBS="-L$(VCPKG_INSTALLED)/lib -lsqlite3 -L$(VCPKG_INSTALLED)/lib -ltiff -L$(VCPKG_INSTALLED)/lib -llzma -L$(VCPKG_INSTALLED)/lib -ljpeg" \
	TIFF_INCLUDE_DIR=-I$(VCPKG_INSTALLED)/include \
	TIFF_CFLAGS=-L$(VCPKG_INSTALLED)/lib \
	SQLITE3_CFLAGS=-I$(VCPKG_INSTALLED)/include \
	SQLITE3_LIBS=-L$(VCPKG_INSTALLED)/lib \
	TIFF_LIBRARY_RELEASE=$(VCPKG_INSTALLED)/lib/libtiff.a

Configure also for GDAL ignores custom curl, sqlite3, hdf5:

gdal.netcore/unix/configuregdal.sh

Lines 23 to 29 in bff4708

	--with-hdf4 \
	--with-hdf5="/usr/lib64" \
	--with-libz=${vcdir} \
	--with-jpeg=${vcdir} \
	--with-expat=${vcdir} \
	--with-libtiff=${vcdir} \
	--with-xerces=${vcdir} \

[Gigas002]

Well done!
In the meanwhile, I've checked the versions in the vcpkg from the latest master branch build (and also attached the links to the repos to easily check for versions):

curl -- 7.72.0 (latest)
sqlite3 -- 3.32.3 (3.33.0 released)
libtiff -- 4.1.0 (latest)
libgeotiff -- 1.4.2-10 (1.6.0 released)
libjpeg-turbo -- 2.0.5 (latest)
libpng -- 1.6.37 (latest)
libiconv -- 1.16 (latest)
zlib -- 1.2.11 (latest)
hdf5 -- 1.12.0 (latest)
libhdfs -- 2019-11-05 (not sure if I located the right library, but I suppose it's outdated, 3.3.0 released on 14.07.2020)
mongo-cxx-driver -- 3.4.0-5 (3.6.0 released)
netcdf-c -- 4.7.3 (4.7.4 released)
xerces-c -- 3.2.3 (latest)
expat -- 2.2.7 (2.2.9 released)
libkml -- 1.3.0-4 (latest?)
proj4 -- 6.3.1 (7.1.0 released)
geos -- 3.8.1 (latest)

Missing:

hdf4 -- 4.2.15

[MaxRev-Dev]

@Gigas002 Good news. After a long break, I'm continuing to implement this feature.
We have already a working windows build and tests are passing!

But currently there is a problem with building PROJ on linux.
All previous configurations were using a system curl library, and I want to link gdal against a version built by vcpkg.
There is a missing curl-config on linux in vcpkg, so I don't know how to make it work.

How it's going - we have an open issue for now: microsoft/vcpkg#14459

gdal.netcore/unix/gdal-makefile

Lines 131 to 142 in 19757e1

	# 1. --with-curl=[path] does not work because curl-config not found
	@cd $(PROJ_ROOT) && ./configure \
	--with-curl=$(VCPKG_INSTALLED_DYNAMIC)/lib/pkgconfig/curl-config \
	--prefix=$(BUILD_ROOT)/proj-build \
	LIBS="-L$(VCPKG_INSTALLED)/lib -lsqlite3 \
	-L$(VCPKG_INSTALLED)/lib -ltiff \
	-L$(VCPKG_INSTALLED)/lib -llzma \
	-L$(VCPKG_INSTALLED)/lib -ljpeg" \
	CURL_INCLUDE_DIR=-I$(VCPKG_INSTALLED_DYNAMIC)/include \
	CURL_LIBRARY="$(VCPKG_INSTALLED_DYNAMIC)/lib/libcurl.so" \
	TIFF_INCLUDE_DIR=-I$(VCPKG_INSTALLED)/include \

[MaxRev-Dev]

Already finished VCPKG integration. Now, everything is controlled by makefiles.
So native dependencies should not conflict with statically linked libraries.
Now we can configure libraries those would have a static or dynamic link with GDAL.
Soon, I will build v3.2.1 of GDAL and update docs.

gdal.netcore/shared/GdalCore.opt

Lines 28 to 41 in f53f3bc

	# we won't use these for now (too many boost deps):
	# -----: libgeotiff libhdfs3 libkml netcdf-c
	VCPKG_REQUIRE=tiff expat xerces-c libpng libiconv zlib
	# windows runtime requires more packages and tools
	# because they are not preinstalled like on linux
	VCPKG_REQUIRE_WIN=$(VCPKG_REQUIRE) sqlite3[tool] curl
	VCPKG_REQUIRE_WIN_STATIC=hdf5
	# the default configuration is static
	# no collisions with sibling libraries in projects
	VCPKG_REQUIRE_UNIX=$(VCPKG_REQUIRE) sqlite3
	VCPKG_REQUIRE_UNIX_DYNAMIC=hdf5 curl

[Gigas002]

Awesome!
Not related to this, but I also think, that we can remove out of support dotnet versons targets (netcoreapp2.2 and netcoreapp3.0)? My guess we should target only LTS releases (2.2, 3.1, upcoming 5.1).

[MaxRev-Dev]

My guess we should target only LTS releases (2.2, 3.1, upcoming 5.1).

Yes, that's a good point about targets. Sure it will be LTS (2.2,3.1) and current (5.0). But what about netstandard2.1 for libraries? It's not deprecated and maybe some libraries would use it.

[Gigas002]

Yes, I suppose it's better to pack netstandard builds at least for a near future.