build: drop support for python 3.6

[dek3rr]

python 3.6 EOL was Dec. 2021
buildbot is running debian 10 which ships with 3.7
meson 0.62+ requires 3.7

Signed-off-by: Doug Kerr dek3rr@gmail.com

[neheb]

and it also breaks compilation on Ubuntu 18.04.

What's the advantage?

[dek3rr]

I'd say the advantage is we don't support legacy scripting and support new versions of meson. Looking forward, not back. It is still possible to upgrade python on ubuntu. I believe EOL exists for a reason, and it eventually becomes more of a pain point to continue support. OpenWRT just had a major release, so I think now is a better time than ever.

[neheb]

I'm not convinced. meson > 0.61 offers no real benefits. I have a lot of experience on that end: https://github.com/mesonbuild/wrapdb/commits?author=neheb

Anyway, I'll give this a run on my Ubuntu 18.04 WSL instance.

[eli-schwartz]

I'm not convinced. meson > 0.61 offers no real benefits. I have a lot of experience on that end

That's arguing from the perspective of a project developer considering to decide on a good minimum requirement in meson.build -- and actually from that perspective it is, literally, irrelevant what you think, because you must use at least whichever version the projects you want to build, support.

The only argument that is relevant here, is whether upgrading Meson is a merge blocker for other packages. And if it is, which one is more important -- the other packages, or supporting Ubuntu 18.04.

(I have no clue what the answer to either of those questions, is.)

[neheb]

No package in OpenWrt currently requires meson >= 0.62.0

As for Ubuntu 18.04, I tested this patch to be working there once python3.8 was installed. One issue which needs to be fixed is: #10325

[eli-schwartz]

No package in OpenWrt currently requires meson >= 0.62.0

Nice, problem solved then. :D

[neheb]

I spoke to soon.

/home/mangix/devstuff/openwrt/staging_dir/host/bin/python3 /home/mangix/devstuff/openwrt/staging_dir/host/bin/meson.py  --native-file /home/mangix/devstuff/openwrt/build_dir/hostpkg/glib-2.70.5/openwrt-native.txt  -Dselinux=disabled -Dlibmount=disabled -Dman=false -Ddtrace=false -Dsystemtap=false -Dsysprof=disabled -Dgtk_doc=false -Dbsymbolic_functions=true -Dforce_posix_threads=true -Dfam=false -Dtests=false -Dinstalled_tests=false -Doss_fuzz=disabled -Dglib_debug=disabled -Dglib_assert=false -Dglib_checks=true -Dlibelf=disabled -Dxattr=false -Ddefault_library=static -Dnls=disabled /home/mangix/devstuff/openwrt/build_dir/hostpkg/glib-2.70.5/openwrt-build /home/mangix/devstuff/openwrt/build_dir/hostpkg/glib-2.70.5/openwrt-build/..
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/mangix/devstuff/openwrt/staging_dir/host/bin/meson.py/__main__.py", line 18, in <module>
  File "/usr/lib/python3.8/pathlib.py", line 4, in <module>
    import ntpath
ModuleNotFoundError: No module named 'ntpath'

@eli-schwartz any idea what this is?

edit: @dekerr dependency on distutils and libpython3.8-stdlib are needed.

[eli-schwartz]

ntpath is a standard library module that provides handling for Windows-based path semantics. It's used for example in this case so that pathlib can create PureWindowsPath objects on Unix systems.

[dhewg]

The only argument that is relevant here, is whether upgrading Meson is a merge blocker for other packages. And if it is, which one is more important -- the other packages, or supporting Ubuntu 18.04.

There's also an argument about meson way to eager to drop support for older versions too soon.

Build machines usually aren't updated as often as workstations. In fact, sometimes it's desired to not use the latest and greatest, like when providing compiled linux binaries and the desire to be binary compatible with older environments. Sometimes it's just not that urgent, because it's, well, just a dumb build box.

Just as with v0.57.0/v3.5 I still think meson underestimates the consequences of such changes. It's only been 3 months between python3.6 going EOL and the first meson release dropping support for it. That's way too soon if the goal is to be the dominating force in the build system world. And that is one of it goals, right? :)

[dek3rr]

There's also an argument about meson way to eager to drop support for older versions too soon.

being a python project, it absolutely makes sense. Packages that it relies on have dropped 3.6 and they can only use the features of the lowest version. That's how you stay a dominating force.

This change should not have an effect on anyone. I'd like to propose some pre-commit hooks in a later PR that would keep our python scripts clean and linted. They rightfully would rely on 3.7+. Python 3.6 is dead so that leaves the question of '"how long do you support something that no one else is using?" I don't quite understand who we're catering to or when the "right" time would be.

[eli-schwartz]

Meson has only dropped support for a python release after that release is end of lifed. That's not unreasonable, because it means you get five years of support. Furthermore, meson benefited tremendously from updating its minimum python requirements -- we extensively took advantage of 3.7 features for e.g. type annotations, dataclasses, stdlib importlib.resources, etc. We are absolutely not under any circumstances ever for any reason no matter what, going to wait until Ubuntu 18.04 is end of lifed in 20 freaking 28 ;) we cannot simply call a moratorium on development for a solid decade.

Fortunately, none of that matters. Ubuntu 18.04 is an LTS release, it doesn't get new features and it doesn't get feature updates to software. Old versions of Meson continue to work.

In fact, Meson does LTS updates too! The latest version of Meson to support python 3.6 got a number of bugfix releases (but not new features, any more than Ubuntu does). Regrettably, Ubuntu does not care what anyone other than Ubuntu does, so they don't update bugfix releases of Meson (LTS or not), but, pip is therefore an option.

I don't advise projects using Meson to require the latest version of Meson anyway -- certainly not for convenience cleanups and (version-guarded) deprecations. It is pretty reasonable and conservative to spend at least another 2 years attempting to preserve compatibility with the version of Meson shipped as part of the Ubuntu 20.04 official repos. That's 0.53, and is fully compatible with Ubuntu 18.04 plus pip. So, no problems there.

Obviously, some projects will have personal reasons to be okay dropping support for Ubuntu 18.04 (the numpy ecosystem for example has a strict support window and recently migrating to Meson does not affect them -- we supported it for longer than they did. They are welcome to use brand new features from Meson 0.63, many of which I wrote specially for them.)

I'm sorry, but I have a very hard time feeling much sympathy for people that insist they need to use an OS that was frozen in amber half a decade ago, and is not even going to be in standard support in a few months, and use a python interpreter that is no longer supported for security vulnerabilities, and want to...

... use bleeding edge feature releases of Meson from a couple months ago that aren't in any distros other than the rolling release ones and the unstable development branches of the stable distros.

I would prefer to focus on providing good support for the already ancient distros that have python 3.7 or 3.8, and for which Meson is already, willingly and happily, avoiding python features that aren't at least 5 years old, and I would -- literally -- be delighted to collaborate with anyone willing to nominate bugfixes (but not feature additions) to the LTS version of Meson.

And as already mentioned, this works out great for OpenWrt, because you don't need Meson 0.62.0 anyway. Projects are avoiding such recent pins anyway, because it doesn't work out of the box (no pip) on Ubuntu 20.04.

It is ethically correct for projects to add new features along with the new requirements that make those new features possible, while sticking to bugfixes for maintenance branches powered by old requirements.

[eli-schwartz]

Furthermore, Ubuntu isn't run by fools, so they backported python 3.8 anyway and you can just use that. Meson even runs CI for 18.04 using exactly that updated python 3.8 package.

Furthermore furthermore, the Meson bump to 3.7 solved major development pain points, but I can't think of anything off the top of my head that would necessitate bumping to 3.8, so we may just skip that because it's not actually constructive to drop support. I'm not in any hurry to do it if it doesn't actually solve real problems. We'll probably want 3.9 though.

[eli-schwartz]

Build machines usually aren't updated as often as workstations. In fact, sometimes it's desired to not use the latest and greatest, like when providing compiled linux binaries and the desire to be binary compatible with older environments.

It is a grave danger to use a build machine that is no longer supported by its vendor, so I would not make Ubuntu 18.04 be critical release building and code signing infrastructure for anything that uses python 3.6, and for anything after 6 months from now.

More generally, your build machine should not be older than the oldest workstations you still support. They should be the last things you upgrade, not the only things you never upgrade, which means... they should still be updated.

But not for another 6 months.

By which time the ecosystem still won't have shifted over to requiring minimum versions of Meson that are too new to run on the build machines you're finally phasing out. Which in turn means you don't even have to upgrade to "the shiniest version of Meson from last month" at that point in time, because... you don't need it yet. :) 🎉

[neheb]

Added PR fixing detection: #10654

If this gets merged, it should be after that one.

[hauke]

I am against merging this, because Ubuntu 18.04 still uses Python 3.6.
We can discuss this again when some package need a more recent meson version or there is any other real need for a more recent Python version.

[dek3rr]

I am against merging this, because Ubuntu 18.04 still uses Python 3.6. We can discuss this again when some package need a more recent meson version or there is any other real need for a more recent Python version.

1. Ubuntu has backported python 3.8
2. I'd like to introduce the use of pre-commit for contributions which requires 3.7
3. 23.XX should be being actively developed in master. Are we really concerned about supporting Ubuntu 18.04 which will be EOL by the time a RC comes along?

From an organizational standpoint, clinging to python 3.6 is an awful hill to die on in 2023

[neheb]

18.04's EOL is in 2028.

[dek3rr]

sorry, EOL was not correct. End of hardware and maintenance. Other points remain.

[dhewg]

Packages that it relies on have dropped 3.6

But is that really the case? mesonbuild/meson#9510 reads more like a nice-to-have

I'm sorry, but I have a very hard time feeling much sympathy for people that insist they need to use an OS that was frozen in amber half a decade ago, and is not even going to be in standard support in a few months, and use a python interpreter that is no longer supported for security vulnerabilities, and want to...

But the point of view that it's just the choice of a single person isn't correct here. Meson is part of OpenWrt, not of the user's distro. If the meson version can't be upgraded due to the Python version, packages requiring those meson versions just can't be used. It sounds liky numpy is such a case?

One can build OpenWrt on LTS releases. That's a feature and more than reasonable, since those are the primary choice for build/CI boxes, like its own buildslaves to build the official releases. I don't agree on the support and security arguments, the very definition of a LTS release is that it's maintained and security flaws gets backported. It may be old, we may not like it, but in the end the build result has nothing to do with the age of the host distro.

Don't get me wrong, I understand wanting to get rid of hacks and workarounds for older Python versions and using the latest and greatest features. But with these choices OpenWrt is forced to choose between dropping certain host distros or running into the situation of being unable to upgrade certain packages - to play the devil's advocate: maybe even security critial releases :P

[neheb]

From IRC:

<jow> people can't comprehend that 10 year old openwrt releases are still actively used

[eli-schwartz]

But the point of view that it's just the choice of a single person isn't correct here. Meson is part of OpenWrt, not of the user's distro. If the meson version can't be upgraded due to the Python version, packages requiring those meson versions just can't be used. It sounds liky numpy is such a case?

I mentioned numpy specifically to say that because they dropped support for python 3.7 at the same time Meson dropped support for 3.6, it's reasonable for them to require versions of Meson that are only compatible with python 3.8

Again, elsewhere, projects are typically being conservative and requiring Meson versions available in Ubuntu 20.04 -- the repository, not the pip install compatibility.

the very definition of a LTS release is that it's maintained and security flaws gets backported. It may be old, we may not like it, but in the end the build result has nothing to do with the age of the host distro.

I agree!

What's also true is that only security flaws get backported, not feature releases.

For example, python 3.6 used to be an LTS release, so only security flaw commits get backported by the python core team.

Ubuntu 18.04 is an LTS release, so only security flaw tags of python get backported by the Ubuntu security team. For example, python 3.6.15 was tagged on September 3, 2021, corresponding to the python 3.6.15 security update in bionic-updates which we can see right here:

Which is pretty cool actually, because I can't imagine how Ubuntu was able to get a preview of python 3.6.15 (modulo the typo for "15") in time to package it way back in 2018.

[neheb]

I just had a thought.

Currently there's no dependency on host pip. However if done, I imagine is can be abused to replace tools/meson. A bit overkill maybe...

[mpratt14]

hi all,

the vastness of this conversation inspired me to do #10800

this is not a serious request to build python instead of requiring it, but just a demonstration that it is possible and not difficult to achieve. And, if the project members decided at some point that the issues presented here are too much to deal with, this alternative will be available.

ideas and comments are appreciated 👍🏼

[eli-schwartz]

This came up again due to #12665.

[ynezz]

Once we branch off 23.05 release, we should merge this.

[neheb]

Don't really get why. Ubuntu 18.04 has 3.8.

[ynezz]

Thanks, merged in 7d0fb2c