Aggressive Dependency Bump for 3.18

[nyalldawson]

QGIS Enhancement: Aggressive Dependency Bump for 3.18

Date 2020/10/08

Author Nyall Dawson (@nyalldawson)

Contact nyall.dawson@gmail.com

Version QGIS 3.18

Summary

The build dependency versions for QGIS have not been raised in many years, and as a consequence we are dragging around a lot of duplicate code designed to work around older dependency versions.

It is proposed that for QGIS 3.18 we perform an aggressive dependency bump and remove support for many older, unsupported library versions.

While this may result in some users on older Linux distributions being unable to compile or run QGIS 3.18, these users will still be able to use the 3.16 LTR release for the duration of its lifetime.

Proposed version changes

PROJ

Drop all support for PROJ < version 6. Proj versions earlier than 6 are completely unsupported by upstream and have critical limitations which can result in misleading transformation results. NO users should be using these versions for any professional work in 2020.

There is 1000s of lines of duplicate code in QGIS master in place to support these older proj versions, and apply a bunch of QGIS specific hacks and custom logic to get things working as well as possible. This code should be removed, so ALL responsibility for projection handling and transformations in QGIS is delegated to the PROJ library (where it belongs)

GDAL

Drop all support for GDAL 2 and require GDAL 3.0 as a minimum version. The GDAL 2.x / proj >= 6 combination is not recommended and suffers from known issues.

This will also allow us to drop many downstream QGIS specific workarounds for bugs since fixed in GDAL upstream.

GEOS

No change to version requirement

Qt

Raise the minimum Qt version up to 5.11 or later. Version < 5.11 have upstream issues resulting in crashes in QGIS print layouts. All versions < 5.15 are unsupported by Qt upstream now, so ideally we'd raise the dependency even further (e.g. to 5.13), but this is likely too aggressive at this stage.

Compiler

Raise the dependency to require a compiler capable of c++17 support, and allow c++17 language additions in the QGIS codebase.

Python / PyQt / sip

We may want to consider also raising the minimum Python/PyQt/SIP versions too.

Affected Files

Everything

Performance Implications

Will be improved 😉

[3nids]

Some outstanding enum issues have been fixed in Qt 5.12 which is LTS (https://bugreports.qt.io/browse/QTBUG-47652, https://bugreports.qt.io/browse/QTBUG-58454).
This currently prevent us from using scoped enums which are way better represented under the PyQGIS docs (they are grouped under their enum name and can have docstrings displayed).

Main backward is that Debian stable is lagging on 5.11.

I would be very highly in favor of requiring Qt 5.12, this would be aligned with moving to Qt6.

[nyalldawson]

Main backward is that Debian stable is lagging on 5.11.

I'd still be supportive of this -- as noted in the proposal, Linux users impacted by their distro running older Qt versions will still have access to the latest LTR release (3.16). Basically we treat 3.18 as an "absolute bleeding edge" release, with the usual disclaimer about recommending that users use the stable LTR release wherever possible.

[lbartoletti]

• Proj : +1
• GDAL : +1 ( >= 3.1.0 ?)
• GEOS : ok
• Qt : I would prefer 5.13 too (there is some issue fixed in qt 5.12 IIRC) but there are stable distributions (debian/centos, etc.. https://repology.org/project/qt/versions) where the version is lower than Qt5.13
• Compiler : c++17
• PyQt : pyqt <= qt ver.
• Sip : qgis build fine with sip4 and sip5. It will be necessary to see with sip6 which can cause installation conflicts for system packagers.

[pcav]

Il 08/10/20 08:01, Nyall Dawson ha scritto:

Main backward is that Debian stable is lagging on 5.11. I'd still be supportive of this -- as noted in the proposal, Linux users impacted by their distro running older Qt versions will still have access to the latest LTR release (3.16). Basically we treat 5.18 as an "absolute bleeding edge" release, with the usual disclaimer about recommending that users use the stable LTR release wherever possible.

agreed fully: Debian stable==QGIS LTR Debian testing/sid==QGIS latest it is rather obvuous that stable is meant for servers, testing for desktop -- Paolo Cavallini www.faunalia.eu - QGIS.org training, support, development on QGIS, PostGIS and more

[PeterPetrik]

as for MacOS nightly packages, we already use the mentioned (or above) versions, e.g. check here https://qgis.org/downloads/macos/nightly/qgis_nightly_master_20201007_122812.deps
(qt is 5.14.2)

[elpaso]

Big +1 here. Especially for GDAL.

[luipir]

I'm almost agree.
Do we have a map of oldest version we can install. I suppose the main problem would be with QT more that gdal or PROJ.
Just two days ago, in university context, asked for a installation problem on a 32b/win7 environment. There are a lot of old and refurbished laptop due to covid19.

[gdt]

I'm concerned about the C++17 requirement. If you mean "allow C++17 that is usable with gcc 7 and clang 5", I think it's just barely ok. If you mean that gcc 7 is unusable and people need 8, I think that's too aggressive.

(The rest seems fine.)

[3nids]

It looks like gcc has support for c++17 since version 7 and clang since version 4
https://gcc.gnu.org/projects/cxx-status.html#cxx17
https://clang.llvm.org/cxx_status.html

[s-m-e]

It appears that QGIS does not have a specific minimum version / dependency of Python 3, see this unanswered e-mail to the dev mailing list. Please correct me if I am wrong. The actual requirement is kind of implicit. At the moment, at least on Linux, it builds against 3.5 and better if my tests are correct. I did not manage to build it against older versions. The oldest supported version of CPython is 3.6, see here. Ubuntu's latest LTS release, 20.04, runs on CPython 3.8. openSUSE's latest LTS, Leap 15.2, still runs on CPython 3.6 - but there a big bump in sight. Apart from language and memory management features, younger versions of CPython have significantly improved Windows support - MS is throwing a lot of resources onto it at the moment.

[gdt]

It looks like gcc has support for c++17 since version 7 and clang since version 4
https://gcc.gnu.org/projects/cxx-status.html#cxx17
https://clang.llvm.org/cxx_status.html

Those say gcc8 and clang5, if one interprets C++17 support to mean full support for the language specification. Hence my question about whether the proposal is for full C++17 support, or "C++17, with the requirement that gcc7 is usable".

[3nids]

Let's try to summarize:

Qt: 5.12
Gdal: 3.1
Python: 3.7 (debian stable has 3.7)
c++ 14 (from discussion above, no strong opinion)
sip: 4.19.21 (ubuntu 20.04 has 4.19.21, debian testing has 4.19.24)

Do we need a formal vote to go on?

[elpaso]

C++ 17 please ...

[nyalldawson]

Can we lower gdal to 3.0 or later? There's no major architectural benefits to blocking 3.0

[gdt]

Similarly, I made a point about C++17 and gcc versions. I would prefer to avoid going to C++17 -- in packaging various things there is often pain from new C++ requirements -- but if we do I think it should be stated as "C++17, that compiles cleanly with gcc 7, with clang 5, and with higher versions of both". Simply stating C++17 means that gcc8 is required, as there are some C++17 features not supported in 7.

All that said, I haven't seen the argument for why stopping building with slightly old compilers is outweighed by use of new features. Not saying it's untrue, but would like to see a cost/benefit claim.

[m-kuhn]

@gdt can you be more precise about your concerns.
Is it particular platforms that come with older compilers (which platforms?) or other pains stemming from C++17? At which point would you think it would be better to move there?

Modern C++ has better support for idiomatic programming and therefore can be used to write better readable code.
From a developer point of view, I'd be happy to be able to have these constructs available.
Also, from a developer point of view, it would be easier to check if construct X is "C++17" rather than to check if construct X is "C++17 compatible with gcc 7 and clang 5".

[gdt]

The issue is systems that have older compilers. gcc 8.1 was only released in May of 2018. From there it takes some time to get into a system's development branch, and then to have a release, and then for users to upgrade. gcc 7.1 was released in May of 2017, and I think we are just about to the point where it is ok to require gcc 7. And yes, people can install a different compiler, but there has been enough of a history of library and ABI incompatibilty that the standard wisdom is "build all C++ programs with the same compiler".

An example is NetBSD 9, which has gcc 7. (The upcoming NetBSD 10 will have newer gcc.) Many GNU/Linux systems are going to be in the same boat. Another example is Ubuntu 18.04 comes with gcc 7. (Yes, I realize it's now after 20.04.) Centos 8 seems to have gcc 8, and Centos 7 is very old. So the question is, is it reasonable to say to someone running 18.04: "Your system is too crufty and you are out of luck". I think that's fair to say to someone running 16.04, but it's not fair for 18.04 yet.

I can also flip the question: People proposing to require gcc 8 have an obligation to explain why that won't cause pain. I haven't seen any analysis of typical user systems here.

I'm curious if you are involved in packaging things i a multi-OS multi-version environment. I have done a lot of that, and my concern about C++ version bumps is based on having experienced a lot of problems from upstream packages requiring new flavors of C++ before they are available in almost all systems. I can see that from your point of view, presumably having chosen to use a system with aggressively updated compilers (a fine choice for you -- I don't mean to criticize it), you'd rather use the new features. I suspect that a very large number of users are on systems with far less aggressive compiler update timelines -- but that almost all developers do not use such systems.

As for saying "is it valid C++17" is easier than "is it ok with gcc 7 and clang 5", I think that the constructs that aren't ok with gcc 7 are rare and it's easy to write that down, basically the table in the gcc docs already linked here. And, in my experience, almost no one writes from the spec. They write things sort of from the spec and if they build on the compiler that is in front of them at the moment it's considered ok. I routinely find non-POSIX constructs that were fairly obviously only tested on Linux. A bit of CI with gcc 7 and clang 5 should be entirely sufficient to catch problems. Probably in about a year it will be reasonable to require 8.

[m-kuhn]

Ubuntu 18.04 will already suffer from other changes in this proposal (e.g. Qt 5.9 < 5.11). I'm not sure about NetBSD 9.

This is summarized by:

While this may result in some users on older Linux distributions being unable to compile or run QGIS 3.18, these users will still be able to use the 3.16 LTR release for the duration of its lifetime.

and

Basically we treat 3.18 as an "absolute bleeding edge" release, with the usual disclaimer about recommending that users use the stable LTR release wherever possible.

By the time this will hit an LTR, Ubuntu 18.04 will be 3.5 to 4 years old. If one is still bound to such a system for stability reasons (a fine choice which I don't mean to criticise, I hope you don't mind if I quote you here 😃 ) one might question himself if the same criteria shouldn't count for the applications running on this system.

[gdt]

FWIW, NetBSD (and the other BSDs) are only the "base" system, what's traditional in Unix, evolved somewhat. So no QT at all. Things beyond the compiler come from a packaging system which is updated for more often. We already have qt 5.15.1 and the next quarter will have 5.15.2.

Your point out LTS users is fair in the abstract. I see it as more nuanced - basically someone wanting to run really old code for stability like RHEL is in one place, and someone who just hasn't updated just after a new stable release is another. Anyway, I think we have reached the point where we understand each other. As always, it's nice to have a polite discussion and thanks for taking the time.

[nyalldawson]

The proj >= 6.0, gdal >= 3.0, python >= 3.7, Qt >= 5.12 and c++ >= 14 requirements have been implemented for QGIS 3.20

[m-kuhn]

Thanks @jef-n !

[3nids]

I add here cmake 3.12 for qgis/QGIS#42497