New package 'meson-py'

[dhomeier]

Packaging the meson build system building on the earlier (closed) attempt from #582 and homebrew recipes.
Built, tested (with a few failures) and used to build scipy 1.9.0rc1 on 12.4/arm64 and 10.14.6.

[eli-schwartz]

Do you actually need to install a polkit file at all? AFAICT fink doesn't have polkit available anyway.

[dhomeier]

Good point, this was left from the initial version; the only other polkit file present there on my system was installed by dpkg.

[dmacks]

I'd been working on meson recently at dmacks:meson2 ("2" because it was my second attempt, not a package-name). It has a few additional patches and documentation of some test-fails. Feel free to adopt this thing, I just really want to see it done and finaly had some free time to work on it.

After investigating the meson project goals and features, I decided to merge it back into a single main package rather than having -pyXX variants. Regardless though, there probably needs to be a main "meson" package that is either the thing itself or a Type:bundle on top of the variants pool (like dmacks@d31ad1a. Otherwise every maintainer has to make a variant decision for a single command-line tool and users might wind up having to install 4 pythons just to build a gnome package.

[eli-schwartz]

This decision is a lot easier for distros and vendors that have a default python which all command-line utilities that happen to be written in python, target by default. I'm not familiar with how fink works though.

I can verify that Meson is supposed to work with any (non-EOL) version of python, and that it simply treats python as:

• an implementation language for writing a CLI utility
• a fallback for find_program('python3'), if it cannot find one in $PATH and none is specified via a --native-file's "binaries" config section then it knows as a last-ditch attempt it can find "some version" via sys.executable; this is primarily used to allow stdlib-only helper scripts that work on Windows, which may not have an existing python.

[dhomeier]

Thanks, I think I had actually gotten a notification about a commit to the old PR that reminded me this was still not available and triggered this PR. Will have a look at the additional fixes for the tests etc.

I agree it makes sense to have at minimum a bundle making a generic meson available; since its functionality is rather independent of the interpreter, there may be no need for versioned packages at all (kind of similar to mercurial I guess, which could certainly get by with a single Python Type). I have not seen any use cases yet where mesonbuild would be imported directly from a Python session (I guess it could come up when e.g. linking it to an IDE).
Just needs a little extra preparation for pythonic packages built with meson then (which I think would automatically fall back to the correct sys.executable with the versioned packages), but for the Scipy build all it took was explicitly setting py_mod.find_installation('%p/bin/python%type_raw[python]') in meson.build.

This decision is a lot easier for distros and vendors that have a default python which all command-line utilities that happen to be written in python, target by default. I'm not familiar with how fink works though.

Settling to Python 3.10 for packages like this, mercurial or libnewt0 would probably a good solution, though a bit of a moving target. I don't think the general Fink system of maintaining several minor versions in parallel is up for discussion at this point.
And I find it actually convenient to have, say all the currently supported versions of my projects, like 3.8-3.11 for Numpy, Astropy etc., available simultaneously on my system. Though having to keep the older versions back to 3.4 alive becomes a real pain when upgrading packages!

[eli-schwartz]

Just needs a little extra preparation for pythonic packages built with meson then (which I think would automatically fall back to the correct sys.executable with the versioned packages), but for the Scipy build all it took was explicitly setting py_mod.find_installation('%p/bin/python%type_raw[python]') in meson.build.

The SciPy build uses a PEP 517 build backend that e.g. when running python3.X -m build to create a wheel, passes the python3.x version as a binary path override, so this should work.

On the other hand, it also wants to have a python metadada dependency on Meson, i.e. depend on a version of Meson packaged for that python version... even though it only runs meson as a CLI tool. It also depends on a python metadata slot for ninja that is packaged on PyPI. Both of these are because pip install cannot invoke a package manager such as apt or pacman or emerge or rpm/dnf, so it decides to just depend on additional internal copies.

[dhomeier]

On the other hand, it also wants to have a python metadada dependency on Meson, i.e. depend on a version of Meson packaged for that python version... even though it only runs meson as a CLI tool. It also depends on a python metadata slot for ninja that is packaged on PyPI. Both of these are because pip install cannot invoke a package manager such as apt or pacman or emerge or rpm/dnf, so it decides to just depend on additional internal copies.

As Fink is building its .deb directly from the install dir (here populated from meson setup build --prefix=%i; ninja -C build; meson install -C build) without building a wheel or invoking pip that does not seem a concern for us. At least, having e.g. only meson-py310 installed, scipy-py39 installs completely and is passing the tests afaics (and vice-versa).

[eli-schwartz]

In that case, the SciPy "dist-info" metadata will be missing, so if users install SciPy through fink, then try to pip install something that depends on fink, pip will not know that SciPy is already installed.

dist-info metadata generation is one of the planned improvements for Meson's python extension building support, it's just not there yet.

That may or may not matter to you. Everything else works with native Meson -- in fact, upstream SciPy's development tooling just uses native Meson to build and install SciPy and run the tests, which works fine.

(I've been working with SciPy to help integrate Meson support.)

[dhomeier]

Thanks for pointing that out, I had noticed there was no egg.info or dist.info installed, but not followed up on that.
It certainly matters, as there is an increasing number of packages Fink does not have the resources to provide as dpkg, so we can only direct users to pip-install those. But at the same time I think it is still very desirable to have pip recognise the already installed core packages, as it does with the setuptools-created debs.

Fortunately the PKG-INFO seems to provide enough data for pip, so as long as we get that from the PyPI sources, it is an easy enough workaround.

I have pushed my WIP package to dhomeier:scipy-1.9 ; if anyone wants to comment on that separately, I can open a draft PR.

[dhomeier]

Added the bundle package now, as there seems sufficient potential demand for the py-versioned packages.

[dmacks]

Python-varianted packaging sounds like a good idea. Thanks everyone for explaining it so clearly!

[dmacks]

nit: no need for fink (>= 0.24.12) BDep

[dhomeier]

No idea how I came up with 0.24.12. But I thought Info2 still required 0.20.0; validation passes without any fink [B]Dep... – maybe assuming there are no more ancient versions in existence!

[dhomeier]

Upstream update came in between; tested on 10.14.6 and 12.4 (arm64+x64_86).
The remaining test failures

FAILED run_unittests.py::AllPlatformTests::test_pkgconfig_gen_escaping - AssertionError: Lists differ: ['-I/usr/with -DLIBFOO', 'spaces/include'] != ['-DLIBF...
FAILED run_unittests.py::NativeFileTests::test_python_module - subprocess.CalledProcessError: Command '['/opt/bin/python3.10', '/private/scratch.noindex/fink....
FAILED run_unittests.py::InternalTests::test_pkgconfig_parse_libs - AssertionError: Lists differ: ['-L/private/tmp/tmpdli2n2ia/2', '-L/privat[34 chars]bar'] ...
FAILED run_unittests.py::FailureTests::test_extraframework_dependency_method - RuntimeError: Configure failed
FAILED run_unittests.py::LinuxlikeTests::test_pkgconfig_uninstalled - subprocess.CalledProcessError: Command '['/opt/bin/ninja', '-w', 'dupbuild=err', '-d', ...

seem in part related to the flag-sorting issue noted, others to looking for a Framework installation. I've also seen failures like

cc -Itest1.p -I. '-I../test cases/common/44 pkgconfig-gen/dependencies' -I/private/scratch.noindex/fink.build/meson-py310-0.63.0-1/meson-0.63.0/tmp4by7txff '-I/private/scratch.noindex/fink.build/meson-py310-0.63.0-1/meson-0.63.0/test -I/private/scratch.noindex/fink.build/meson-py310-0.63.0-1/meson-0.63.0/tmp4by7txff' '-I/private/scratch.noindex/fink.build/meson-py310-0.63.0-1/meson-0.63.0/test -DLIBFOO' -fcolor-diagnostics -Wall -Winvalid-pch -O0 -g 'cases/common/44 pkgconfig-gen' 'cases/common/44 pkgconfig-gen' -MD -MQ test1.p/main.c.o -MF test1.p/main.c.o.d -o test1.p/main.c.o -c '../test cases/common/44 pkgconfig-gen/dependencies/main.c'
clang: error: no such file or directory: 'cases/common/44 pkgconfig-gen'
clang: error: no such file or directory: 'cases/common/44 pkgconfig-gen'

indicating the trouble handling path names with spaces.
Could not find any details on the ninja failure.

[dhomeier]

Tested the new release with 3 failures left in the main (pytest) suite, second (cmake tests) part seems to be missing an appleframeworks dependency. Also tested to successfully build scipy-py 1.10.0.