Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proj + libgdal: update and rebuild dependants #31687

Closed
wants to merge 11 commits into from

Conversation

ar-jan
Copy link
Contributor

@ar-jan ar-jan commented Jun 27, 2021

General

Have the results of the proposed changes been tested?

  • I use the packages affected by the proposed changes on a regular basis and confirm this PR works for me
    I use PostGIS regularly and this works for me.
  • I generally don't use the affected packages but briefly tested this PR
    I generally don't use the other packages which have to be rebuilt against proj and/or gdal, but I did test all of them briefly.

Motivation

  • These updates are in preparation for adding QGIS.

@ar-jan
Copy link
Contributor Author

ar-jan commented Jun 28, 2021

Hello @Chocimier, do you have any insight into why postgis builds fail the checks in this PR? It's unrelated to the updates of gdal and proj, I see the same thing when just doing a compile locally on master branch. The build against updated gdal/proj (without checks) seems to function correctly locally.

Not sure if missing xsltproc is the actual problem, but adding libxslt to the template did not resolve that error.

Copy link
Member

@Johnnynator Johnnynator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did give it a quick review. Also pls do no add Packages that do list/change maintainership to someone that isn't you (without their explicit consent)

srcpkgs/libgdal/template Outdated Show resolved Hide resolved
srcpkgs/libgdal/template Outdated Show resolved Hide resolved
srcpkgs/libgdal/template Outdated Show resolved Hide resolved
srcpkgs/libgdal/template Outdated Show resolved Hide resolved
srcpkgs/libkml/template Outdated Show resolved Hide resolved
srcpkgs/libgdal/template Outdated Show resolved Hide resolved
srcpkgs/libgdal/template Outdated Show resolved Hide resolved
@Chocimier
Copy link
Member

Disabled failing tests for now, please rebase.

@ar-jan
Copy link
Contributor Author

ar-jan commented Jun 29, 2021

I've changed/reverted the maintainer lines where applicable. I'd been in touch with Nyx70 previously and he gave thumbs up on continuing with his work, but wasn't explicit about maintainership.

Re archs="~aarch* ~armv*" for libgdal: I tried building for armv7l, and it failed with:

x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-mfpu=vfpv3'
x86_64-unknown-linux-gnu-gcc: error: unrecognized command-line option '-mfloat-abi=hard'
error: command '/usr/bin/x86_64-unknown-linux-gnu-gcc' failed with exit code 1
make: *** [GNUmakefile:73: build] Error 1
�[1m�[31m=> ERROR: libgdal-3.2.3_1: post_build: 'make ${makejobs} PYTHON=python3 ${makejobs}' exited with 2
�[m�[1m�[31m=> ERROR: in post_build() at srcpkgs/libgdal/template:45

Is this a matter of removing those flags, and if so, how to do that?

Finally there's one more package to be rebuilt against libgdal and proj, grass. The currently packaged version is apparently not working #29209; I built the new grass version 7.8.5 but it still has non-working GUI. So maybe that can be done separately.

I've resolved the xlint license issue for osg, but if I push that it'll trigger another build, maybe that's not desirable?

@sgn
Copy link
Member

sgn commented Jun 30, 2021

diff --git a/srcpkgs/libgdal/template b/srcpkgs/libgdal/template
index 2c6f38a328..f71d947b0a 100644
--- a/srcpkgs/libgdal/template
+++ b/srcpkgs/libgdal/template
@@ -2,8 +2,6 @@
 pkgname=libgdal
 version=3.2.3
 revision=1
-# aarch & arm currently failing
-archs="~aarch* ~armv*"
 wrksrc="gdal-${version}"
 build_style=gnu-configure
 configure_args="
@@ -39,20 +37,41 @@ build_options_default="kml"
 if [ -z "$CROSS_BUILD" ]; then
 	makedepends+=" hdf5-devel"
 fi
+CFLAGS="-pthread -I${XBPS_CROSS_BASE}/${py3_inc}"
+LDFLAGS="-L${XBPS_CROSS_BASE}/${py3_lib}"
 
 post_build() {
+	if [ "$CROSS_BUILD" ]; then
+		export PYPREFIX="$XBPS_CROSS_BASE"
+		export PYTHONPATH=${XBPS_CROSS_BASE}/${py3_lib}
+		for f in ${XBPS_CROSS_BASE}/${py3_lib}/_sysconfigdata_*; do
+			f=${f##*/}
+			export _PYTHON_SYSCONFIGDATA_NAME=${f%.py}
+		done
+	fi
+	export LDSHARED="${CC} $CFLAGS -shared $LDFLAGS"
+
 	rm -f swig/python/*_wrap.cpp
 	make -C swig/python generate
 	cd swig/python
-	make ${makejobs} PYTHON=python3 ${makejobs}
+	python3 setup.py build
 }
 
 post_install() {
 	vinstall gdal.pc 644 usr/lib/pkgconfig
 	vlicense LICENSE.TXT
-	# python modules
+	if [ "$CROSS_BUILD" ]; then
+		export PYPREFIX="$XBPS_CROSS_BASE"
+		export PYTHONPATH=${XBPS_CROSS_BASE}/${py3_lib}
+		for f in ${XBPS_CROSS_BASE}/${py3_lib}/_sysconfigdata_*; do
+			f=${f##*/}
+			export _PYTHON_SYSCONFIGDATA_NAME=${f%.py}
+		done
+	fi
+	export LDSHARED="${CC} $CFLAGS -shared $LDFLAGS"
+
 	cd swig/python
-	make PYTHON=python3 DESTDIR=${DESTDIR}/ install
+	python3 setup.py install --prefix=/usr --root=$DESTDIR
 }
 
 libgdal-tools_package() {
@@ -79,9 +98,8 @@ python3-gdal_package() {
 	depends="${sourcepkg}>=${version}_${revision}"
 	short_desc+=" - Python3 bindings"
 	pkg_install() {
-		vmove usr/bin/*.py
+		vmove "usr/bin/*.py"
 		vmove "usr/lib/python*"
-		vlicense LICENSE.TXT
 		vdoc swig/python/README.rst
 		vmkdir usr/share/python3-gdal
 		vcopy swig/python/samples usr/share/python3-gdal/examples

@ar-jan
Copy link
Contributor Author

ar-jan commented Jun 30, 2021

Thanks @sgn!

Re OpenOrienteering-Mapper, tests passed locally, it looks like the following failure is specific to the test runner. How best to handle that?

2021-06-30T02:21:26.4698109Z Start 28: sensors_t
2021-06-30T02:21:26.9903615Z 28/28 Test #28: sensors_t ..........................***Failed 0.52 sec
2021-06-30T02:21:26.9905394Z QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
2021-06-30T02:21:26.9906423Z ********* Start testing of SensorsTest *********
2021-06-30T02:21:26.9908172Z Config: Using QtTest library 5.15.2, Qt 5.15.2 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 10.2.1 20201203), unknown unknown
2021-06-30T02:21:26.9909377Z PASS : SensorsTest::initTestCase()
2021-06-30T02:21:26.9910507Z PASS : SensorsTest::fakePositionSourcePluginTest()
2021-06-30T02:21:26.9911914Z PASS : SensorsTest::fakePositionSourceSimulatedTest()
2021-06-30T02:21:26.9913326Z PASS : SensorsTest::nmeaPositionSourcePluginTest()
2021-06-30T02:21:26.9914999Z QDEBUG : SensorsTest::nmeaPositionSourceSimulatedTest() NmeaPositionSource device: /dev/null
2021-06-30T02:21:26.9917801Z QDEBUG : SensorsTest::nmeaPositionSourceSimulatedTest() NmeaPositionSource device: /builddir/mapper-0.9.5/test/data/sensors/nmea.txt
2021-06-30T02:21:26.9920082Z QDEBUG : SensorsTest::nmeaPositionSourceSimulatedTest() NmeaPositionSource device: /tmp/sensors_t.EcPCyR
2021-06-30T02:21:26.9922022Z FAIL! : SensorsTest::nmeaPositionSourceSimulatedTest() Compared values are not the same
2021-06-30T02:21:26.9923646Z Actual (int(source->error())) : 3
2021-06-30T02:21:26.9924385Z Expected (int(QGeoPositionInfoSource::AccessError)): 0
2021-06-30T02:21:26.9925350Z Loc: [../test/sensors_t.cpp(150)]
2021-06-30T02:21:26.9926306Z PASS : SensorsTest::powershellPositionSourcePluginTest()
2021-06-30T02:21:26.9927534Z SKIP : SensorsTest::powershellPositionSourceLiveTest() Powershell not available
2021-06-30T02:21:26.9928623Z Loc: [../test/sensors_t.cpp(169)]
2021-06-30T02:21:26.9929544Z PASS : SensorsTest::powershellPositionSourceSimulatedTest()
2021-06-30T02:21:26.9930561Z PASS : SensorsTest::cleanupTestCase()
2021-06-30T02:21:26.9931272Z Totals: 7 passed, 1 failed, 1 skipped, 0 blacklisted, 504ms
2021-06-30T02:21:26.9931955Z ********* Finished testing of SensorsTest *********
2021-06-30T02:21:26.9932756Z CMake Error at sensors_t-RUN.cmake:35 (message):
2021-06-30T02:21:26.9933323Z Test sensors_t failed: 1
2021-06-30T02:21:26.9949629Z 96% tests passed, 1 tests failed out of 28
2021-06-30T02:21:26.9950377Z Total Test time (real) = 19.33 sec
2021-06-30T02:21:26.9951106Z The following tests FAILED:
2021-06-30T02:21:26.9951982Z 28 - sensors_t (Failed)
2021-06-30T02:21:26.9952450Z Errors while running CTest
2021-06-30T02:21:26.9952979Z FAILED: CMakeFiles/test.util
2021-06-30T02:21:26.9953967Z cd /builddir/mapper-0.9.5/build && /usr/bin/ctest --force-new-ctest-process
2021-06-30T02:21:26.9954852Z ninja: build stopped: subcommand failed.
2021-06-30T02:21:26.9956018Z => ERROR: OpenOrienteering-Mapper-0.9.5_2: do_check: '${make_cmd} ${make_check_args} ${make_check_target}' exited with 1
2021-06-30T02:21:26.9957145Z => ERROR: in do_check() at common/build-style/cmake.sh:116

@Johnnynator
Copy link
Member

You can add

do_check() {
	cd build
	ctest -E 'sensors_t'
}

which disables that specific test

@ar-jan ar-jan force-pushed the gdal-proj branch 3 times, most recently from 881b60d to 77c29c0 Compare July 2, 2021 10:05
@ar-jan
Copy link
Contributor Author

ar-jan commented Jul 2, 2021

Apparently, the Grass GUI currently packaged is not functional (#29209). The updated version compiles and starts successfully, but in my testing the GUI disappears as soon as the mouse pointer enters its area. I do not see any error message. Any clues how to find the cause? If not perhaps it can be merged anyway since the GUI is already not functional in the current state and otherwise the library versions are a problem.

@ar-jan ar-jan requested a review from Johnnynator July 2, 2021 17:10
@ar-jan ar-jan changed the title [WiP] Update libgdal and proj and rebuild against them Update libgdal and proj and rebuild against them Jul 2, 2021
@ar-jan
Copy link
Contributor Author

ar-jan commented Jul 5, 2021

The Grass GUI issue is due to to wxPython4 with Python 3.9 which segfaults.

OSGeo/grass#1261
OSGeo/grass#1123

This issue is fixed in wxPython4-4.1.1 which requires wxWidgets 3.1, but I assume it's not desirable to update to the 3.1 branch since it isn't stable like the current 3.0 series. I tried with a backported patch to wxPython4 but still had the same issue.

@ericonr
Copy link
Member

ericonr commented Aug 12, 2021

This issue is fixed in wxPython4-4.1.1 which requires wxWidgets 3.1, but I assume it's not desirable to update to the 3.1 branch since it isn't stable like the current 3.0 series.

Not only that, it doesn't even work for all our applications...

Do you have any suggestions?

@ar-jan
Copy link
Contributor Author

ar-jan commented Aug 12, 2021

That first part was probably wrong -- what seems to be the underlying bug is fixed in that version, but I've tried with a backported patch which does work for Fedora, but without success. So either I'm doing something wrong in using the patched wxPython, or maybe something else is wrong (but that would be weird given that it fails in exactly the same way as described there).

@ericonr
Copy link
Member

ericonr commented Aug 12, 2021

What steps did you follow?

@ar-jan
Copy link
Contributor Author

ar-jan commented Aug 16, 2021

I built wxPython4 with a backport of the patch as in this branch, and built grass 7.8.5.

Then installed wxPython4 and grass from this local build, then ran grass78.

I supposed it's only a runtime dependency, but to be sure I also built wxPython (v3.0) with the same patch backported, since wxPython-devel is used in building grass, then re-built grass. Same result.

I just switched to a new installation using i3 and I'm now seeing an additional error that I didn't see before: "(wxgui.py:24024): Gtk-CRITICAL: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkSpinButton" (maybe unrelated to the GUI crashing).

@ar-jan ar-jan force-pushed the gdal-proj branch 3 times, most recently from 4f920c9 to 13c821f Compare February 15, 2022 00:35
@ar-jan
Copy link
Contributor Author

ar-jan commented Feb 15, 2022

The grass package held things back six months ago, otherwise everything looked good.

I've given it another try, but it doesn't look hopeful: Python 3.10 + wxPython < 4.1 just doesn't seem to work. I've used grass-7.8.7RC1, which contains a workaround for a wxPython issue as well as fixes related to Python 3.10.
I've also backported patches from wxPython 4.1 to 4.0.7 (following Fedora here, here, and here)

With these changes the grass gui still doesn't work, and it looks like it is simply not feasible with wxPython 4.0. I've included the grass and wxPython4 commits hoping someone can sanity-check this.

If there's no progress with grass, I propose to either:

1. Include 7.8.7RC1 as is: the --text mode is functional (I think, I rarely use grass), and since the curently packaged grass gui doesn't work either it's not a regression afaict.
2. Remove the grass package.

In both cases the wxPython4 update can be left out -- unless these Python 3.10-related patches are helpful in other places?

Edit: I've simplified the PR by only rebuilding the current version of grass, and omitting libspatialite/libkml/librttopo for a separate PR. Grass looks as functional as before.

@ar-jan ar-jan force-pushed the gdal-proj branch 5 times, most recently from 64aa285 to bf26193 Compare February 15, 2022 22:50
@ar-jan
Copy link
Contributor Author

ar-jan commented Apr 25, 2022

Anyone have any ideas of why the crossbuild fails? @sgn maybe? It was working before. It seems that the swig build is now attempted in the do_build step, while it should be occurring in the custom post_build step.

@github-actions
Copy link

Pull Requests become stale 90 days after last activity and are closed 14 days after that. If this pull request is still relevant bump it or assign it.

@github-actions github-actions bot added the Stale label Jul 25, 2022
@Chocimier
Copy link
Member

I could take a look on cross compilation, but logs expired. Please rebase.

@ar-jan
Copy link
Contributor Author

ar-jan commented Aug 6, 2022

I'm currently without a Void system available, will rebase later. @sgn was also going to look at it (via #36816).

@ar-jan
Copy link
Contributor Author

ar-jan commented Oct 25, 2022

I updated the branch. @Chocimier or @sgn want to take a look at libgdal cross compilation?

(The other builds are failing because postgis-postgresql12 install is attempted).

@ar-jan
Copy link
Contributor Author

ar-jan commented Oct 29, 2022

I've updated the libgdal template like so (not pushing to avoid needless 4 hour CI build):

Updated libgdal-3.4.3 template
# Template file for 'libgdal'
pkgname=libgdal
version=3.4.3
revision=1
wrksrc="gdal-${version}"
build_style=gnu-configure
build_helper="python3 numpy"
configure_args="--with-expat=yes --with-hdf5=yes --with-kml=no
 --with-liblzma=yes --with-opencl=yes --with-pg=yes --with-podofo=yes
 --with-python=yes --with-spatialite=yes --with-sqlite3=yes --with-webp=yes
 --with-zstd=yes"
hostmakedepends="gettext-devel json-c-devel pkg-config python3-numpy swig"
makedepends="boost-devel expat-devel freexl-devel geos-devel jasper-devel
 json-c-devel libcurl-devel libopenexr-devel libopenjpeg2-devel libpodofo-devel
 libqhull-devel libwebp-devel libxml2-devel libzstd-devel netcdf-devel
 ocl-icd-devel opencl2-headers pcre2-devel proj-devel python3-devel
 sqlite-devel postgresql-libs-devel"
short_desc="Geospatial Data Abstraction Library"
maintainer="Orphaned <orphan@voidlinux.org>"
license="MIT"
homepage="https://gdal.org/"
distfiles="https://github.com/OSGeo/gdal/releases/download/v${version}/gdal-${version}.tar.gz"
checksum=7244962628c82921b4a4903dbb721c7222b7d82ed5928fef55a52f87f68ad1fe
subpackages="python3-gdal libgdal-devel libgdal-tools"

if [ -z "$CROSS_BUILD" ]; then
	makedepends+=" hdf5-devel"
fi

post_install() {
	vinstall gdal.pc 644 usr/lib/pkgconfig
	vlicense LICENSE.TXT
}

libgdal-tools_package() {
	depends="${sourcepkg}>=${version}_${revision}"
	short_desc+=" - tools"
	pkg_install() {
		vmove usr/bin
	}
}

libgdal-devel_package() {
	depends="${sourcepkg}>=${version}_${revision}"
	short_desc+=" - development files"
	pkg_install() {
		vmove usr/bin/gdal-config
		vmove usr/include
		vmove usr/lib/pkgconfig
		vmove usr/lib/*.a
		vmove usr/lib/*.so
	}
}

python3-gdal_package() {
	depends="${sourcepkg}>=${version}_${revision}"
	short_desc+=" - Python3 bindings"
	pkg_install() {
		vmove "usr/bin/*.py"
		vmove "usr/lib/python*"
		vdoc swig/python/README.rst
		vmkdir usr/share/python3-gdal
	}
}

But it looks like starting with swig/python/setup.py the crossbase is no longer used correctly:

make[1]: Entering directory '/builddir/gdal-3.4.3/swig'
for dir in python ; do (cd $dir; make build) || exit; done
make[2]: Entering directory '/builddir/gdal-3.4.3/swig/python'
...
python setup.py build
WARNING: "CXX=/bin/sh /builddir/gdal-3.4.3/libtool --mode=compile --silent --tag=CXX armv7l-linux-gnueabihf-c++" was defined in the environment and contains more than one word. Unsetting it since that is incompatible of distutils
WARNING: "CC=/bin/sh /builddir/gdal-3.4.3/libtool --mode=compile --silent --tag=CC armv7l-linux-gnueabihf-gcc -pthread -O2 -pipe -march=armv7-a -mfpu=vfpv3 -mfloat-abi=hard -I/usr/armv7l-linux-gnueabihf/usr/include/python3.10 -I/usr/armv7l-linux-gnueabihf/usr/include -L/usr/armv7l-linux-gnueabihf/usr/lib/python3.10 -L/usr/armv7l-linux-gnueabihf/usr/lib" was defined in the environment and contains more than one word. Unsetting it since that is incompatible of distutils
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
...
creating build/temp.linux-x86_64-cpython-310/extensions
...
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv7-a -mfpu=vfpv3 -mfloat-abi=hard -g -I/usr/armv7l-linux-gnueabihf/usr/include -fdebug-prefix-map=/builddir/Python-3.10.8=. -fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv7-a -mfpu=vfpv3 -mfloat-abi=hard -g -I/usr/armv7l-linux-gnueabihf/usr/include -fdebug-prefix-map=/builddir/Python-3.10.8=. -fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv7-a -mfpu=vfpv3 -mfloat-abi=hard -I/usr/armv7l-linux-gnueabihf/usr/include/python3.10 -I/usr/armv7l-linux-gnueabihf/usr/include -I/usr/armv7l-linux-gnueabihf/usr/lib/python3.10/site-packages/numpy/core/include -I/usr/armv7l-linux-gnueabihf/usr/include -fdebug-prefix-map=/builddir/gdal-3.4.3=. -Wall -Wextra -Winit-self -Wunused-parameter -Wmissing-prototypes -Wmissing-declarations -Wformat -Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Wmissing-include-dirs -Werror=vla -Wdate-time -Wnull-dereference -Wduplicated-cond -Wfloat-conversion -DGNM_ENABLED -iquote /builddir/gdal-3.4.3/port -I/usr/armv7l-linux-gnueabihf/usr/include/openjpeg-2.5 -DGDAL_COMPILATION -fPIC -I../../port -I../../gcore -I../../alg -I../../ogr/ -I../../ogr/ogrsf_frmts -I../../gnm -I../../apps -I/usr/include/python3.10 -I/usr/lib/python3.10/site-packages/numpy/core/include -I/builddir/gdal-3.4.3/include -c extensions/ogr_wrap.cpp -o build/temp.linux-x86_64-cpython-310/extensions/ogr_wrap.o -I/builddir/gdal-3.4.3/port -I/builddir/gdal-3.4.3/gcore -I/builddir/gdal-3.4.3/alg -I/builddir/gdal-3.4.3/gnm -I/builddir/gdal-3.4.3/ogr -I/builddir/gdal-3.4.3/ogr/ogrsf_frmts -I/builddir/gdal-3.4.3/frmts/vrt -I/builddir/gdal-3.4.3/apps
gcc: error: unrecognized command-line option '-mfpu=vfpv3'
gcc: error: unrecognized command-line option '-mfloat-abi=hard'

I think this is because CC and CXX are unset? https://github.com/OSGeo/gdal/blob/v3.4.3/gdal/swig/python/setup.py#L22-L42.

Is that right? How to fix?

@classabbyamp
Copy link
Member

libgdal was updated in #40225, can you rebase for the proj update?

@ar-jan ar-jan closed this Nov 5, 2022
@ar-jan
Copy link
Contributor Author

ar-jan commented Nov 5, 2022

(easier to make a new branch)

@ar-jan ar-jan mentioned this pull request Nov 5, 2022
@ar-jan ar-jan deleted the gdal-proj branch November 5, 2022 17:56
@classabbyamp classabbyamp mentioned this pull request Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

libgdal - Please update package and enable python bind
6 participants