Port build system to Meson

[QuLogic]

PR summary

Now that NumPy and SciPy have spent some time on Meson, it's not too bad to move our build system over to Meson as well. This PR also implements the move to pyproject.toml from #23829. I have added some documentation with some key points for migration, though most are edge cases that most developers shouldn't have to worry about.

The main change is that editable installs should disable build isolation:

$ pip install --no-build-isolation --editable .

This is because pyproject.toml-based install would normally isolate builds and extensions are built in the isolated environment, which goes against the idea of an editable install. (You will also need to install build requirements in your target environment, but I think that's not new anyway.)

The other downside is that all files need to be listed manually in most meson.build. However, this also has a side benefit of not accidentally installing extra files (e.g., I get several temporary *.gcno files and the whole node_modules directory if they happen to exist on main.)

The upside is that compilation is written in a very standard way. Both FreeType and Qhull are subprojects using manually written meson.build but those are just the same as the files in the main source. Qhull's build is a simple port of what we did in setupext.py to meson.build. FreeType's build is based on the 2.9.1 version in Meson's wrapdb, but dropping some of the extra install bits we won't use and rolled back to 2.6.1. For 2.11.1, we use the Meson build system already part of FreeType. When we move forward on FreeType versions, we can drop the patched-in build and use the upstream build system.

The other upside of Meson+meson-python builds is that compilation is a) faster with better parallelism since the entire set of files can be seen by the build system, and b) can be done at import time in an editable install. The latter means that you will no longer need to re-install when editing C(++) files (unless you switch to an older commit.)

Closes #23829

PR checklist

• "closes #0000" is in the body of the PR description to link the related issue
• new and changed code is tested
• [n/a] Plotting related features are demonstrated in an example
• New Features and API Changes are noted with a directive and release note
• Documentation complies with general and docstring guidelines

[QuLogic]

And looking back at #23829, I forgot about adding the dev extra.

[WeatherGod]

This should have an if __name__ == '__main__': guard

[QuLogic]

OK, I think we're at a good point now. A couple of issues had to be cleared up:

• I accidentally forgot to install the macosx backend, which made it unavailable in tests, so auto-picking in subprocesses chose TkAgg, which is still broken on Azure. I have fixed the backend here, but also made the subprocess explicitly use Agg where possible in Fix flaky CI tests #26680. Unfortunately, I don't think we can fix test_lazy_auto_backend_selection as its point is to not have a backend set.
• In some CI, we use the backend override from mplsetup.cfg; I've replicated that using the Meson equivalent, but I wonder if we should do that everywhere.
• Azure and AppVeyor have a ton of compilers preinstalled in addition to MSVC. Meson picks the non-MSVC option by default, so I have explicitly passed the option to use MSVC in CI. Since we don't specifically hard-require MSVC, I have not set that option in our config, and I expect most developers do not have multiple compilers on Windows, but it is a gotcha on CI.

There are also a couple followups that need to be considered eventually:

• Versions are not dynamically generated at build time. At runtime, we use setuptools-scm, so it is fine for editable installs. However, this does mean that wheels all use 3.9.0.dev0 as version, which reduces utility of the nightly wheels slightly. I'm not sure if we can hack around this post-build. Upstream is working on a better API for this: Support dynamic version generation, including from git tags and with git commit hashes mesonbuild/meson-python#159
• I don't think all license files are installed the same way as setuptools. I'm not sure if we have any easy way to fix that right now. There is an uptsream issue for this as well: Allow license to be listed in the dynamic section mesonbuild/meson-python#270

There isn't a huge change for developers, and you might be able to glean that from the CI changes (though most changes are extraneous for local work.) There may be a bit more change for distributions, but they might be used to that from NumPy/SciPy's existing change. I will write up some docs for that next week.

[QuLogic]

OK, there is actually one stall on the macOS arm64 builds, which I've reported upstream at mesonbuild/meson-python#468

[ksunden]

I have found a solution that appears to work regarding getting stubtest running with editable install, though not sure its actually a good solution...

If you add the $REPO_ROOT/lib to the mesonpy-generated matplotlib-editable.pth file in site-packages, stubtest is able to run and give accurate results. (One slight change from setuptools editable install is that we had unused allowlist entries for untyped subpackages, but that is pretty trivial)

As far as I can tell it does not affect the special handling set up by mesonpy in _matplotlib_editable_loader, but does allow mypy to connect the stubs to the runtime.

All that said, while it may be good enough for a controlled CI environment, I don't like that as a solution for developers who may wish to run stubtest on their machine (note that mypy alone will work fine, just not with -p or -m, you must give it the lib/matplotlib path... stubtest requires runtime, so is more restricted) In my testing my editor integration still worked fine without any changes.

Perhaps this is a safe enough change and we should lobby/PR upstream to include it to support the mypy usecase.

The other option which seems to also be broken right now, but I don't think should be a problem, is to do a non-editable install... this is not great for the developer machine use case either, but at least does not involve editing generated files...

Currently I'm getting:

error: matplotlib failed to import, InvalidVersion: Invalid version: '"3.9.0.dev0"'

Which is a weird error on a couple fronts, firstly being why is version needed for importing in the first place?, secondly being why is that an invalid version?

Investigation has revealed that stubtest is doing (the __import__ equivalent of) from matplotlib import *, and that is what is triggering the "InvalidVersion". Testing on a REPL confirms that while import matplotlib is fine, trying to get __version_info__ (including a * import) breaks. The problem appears to be that there are quotes on __version__.

In summary:

• I think after sorting out __version_info__ problems probably the best short term path for CI is to use non-editable install for the stubtest job
• If that doesn't work, editing the pth file should
• May be worth talking to upstream about this usecase and seeing if the addition to the pth file can be included there, which would help for developer machines without having to edit a generated file.

[oscargus]

This seems to be OK now? To some extent it may be better to try and get this in so that there is ample time before the 3.9-release to find any issues, but I am still a bit worried to merge it myself...

Well, what is the worst that can happen?

[greglucas]

To some extent it may be better to try and get this in so that there is ample time before the 3.9-release to find any issues, but I am still a bit worried to merge it myself...

Well, what is the worst that can happen?

I would not be worried. You had another approving review and merged it as the maintainer's workflow says. As you say, this will give a good opportunity to see if anything does crop up rather than waiting until the last minute.

[QuLogic]

This removes special casing for AIX and IBM i OS, but the FreeType build is using an included Meson system instead of their own old autotools-based build.

@ayappanec @zheddie can you confirm whether the main branch still builds for you?

[ayappanec]

@KamathForAIX Can you confirm whether the main branch builds fine in AIX ?

[KamathForAIX]

Hi @QuLogic ,

So in AIX gcc with LTO option is not supported. So in meson.build file we should not use it as a default option if the target is AIX. If we can write a logic like that it would be great.

Secondly, in the matplotlib/tools/generate_matplotlibrc.py I had to change !/usr/bin/env python to /usr/bin/env python3.9..

With the two changes mentioned above we are able to build in AIX. Kindly let me know what we can do regarding the same.

[tacaswell]

Does !/usr/bin/env python3 work (rather than python3.9)?

[KamathForAIX]

@tacaswell Yes it works

[QuLogic]

So in AIX gcc with LTO option is not supported. So in meson.build file we should not use it as a default option if the target is AIX. If we can write a logic like that it would be great.

If that is not supported, then this should be handled upstream in Meson. If they silently ignore b_lto=true, then we will have to do nothing. If they error out, then we should see if we can conditionalize it.

[KamathForAIX]

Hi @QuLogic ,

Sure. We will work this out with the meson community members. No worries. Thank you for connecting with AIX for the meson build system test for Matplotlib. If there is anything update or help needed we will get back to you.

Is there anything else we can do for Matplotlib community from AIX? Kindly let us know.

Have a nice day ahead.

Regards,
Aditya Kamath.

[tacaswell]

@KamathForAIX I do not know of any CI service that offer AIX workers so we are a bit blind to if we accidentally break anything. If you could have a monthly cron-job that tries to build Matplotlib and run the tests and report any failures up to us that would be super helpful!

[KamathForAIX]

Hi @tacaswell

Sure, we will set a CI for matplotlib and run the tests.