setup.py implementation

[julian-smith-artifex-com]

[This discussion is branched off #2628.]

@dvzrv wrote in #2628 (comment)

Similar to swig, meson offers a PEP517 build backend. Using it has the upside, that you are using a tool that is actually made for building C/C++ software.
Scripting all of this in setup.py (and pipcl.py) leads to a huge script, that is brittle and (currently) lacks all the common use-cases (e.g. passing in of build flags).

PyMuPDF's setup.py and pipcl.py really don't "lack all the common use-cases", and i'm not sure it's helpful to say so.

As i've already said, they follow the Python packaging spec described in PEP-517 etc, so work with pip and other build front ends, and they also support legacy command-line usage.

Adding in support for passing in compile/include/link flags is trivial to add, and working in my tree.

Most of that is not really a build system, and i'm not sure that something like meson or cmake would be better than the pure Python code in setup.py and pipcl.py.

All of that is part of a good build system! Please have a look at the generic integration for subprojects and the wrap dependency system.

From those links, it looks like Meson can only deal with sub-projects that also use Meson.

You could suggest to the MuPDF team that they switch to Meson i guess, but i don't think it's likely to be agreed.

Meson's declarative syntax is easy to read, has standardized functionality, powerful integration for dependencies, is tested across many platforms and therefore by all means better than custom Python code written for a single project.

I'm sure meson is great.

But as far as i can tell, it has no support for SWIG, so it is a complete non-starter for MuPDF's Python bindings and PyMuPDF.

The intention is that setup.py and pipcl.py can be understood by reading the code and documentation. It looks like this might not be the case though. Can you describe what's lacking for you? I'd be happy to add something, e.g. to the top of setup.py, or in a separate document, that's more aligned with what you're primarily looking for.

Sorry, but having to write thousands of lines of custom Python code to build a project (and forcing others to read it) sounds a lot like NIH to me. :( People know how to read a CMakeLists.txt or a meson.build file (all keywords are documented, there is no ambiguity).

I understand the concern here, and i agree that using an existing tool would be preferable. But the prime requirement is that the build has to actually work, and i couldn't find anything that could cope with the requirements for building PyMuPDF.

So we've opted for a new generic Python packaging library (pipcl.py) and a customised setup.py.

With a custom setup.py and accompanying scripts they have to read the entire thing to even figure out what is overridden where, etc. (because nothing in it will adhere to any standard whatsoever). As a downstream packager with hundreds of packages to maintain, this is a huge concern to me. I simply do not have the (unpaid) time to read through all of this to figure out in which code path my use-case probably is in.

There is no need to read all of the code.

• For someone who is doing a standard build of PyMuPDF, all you need to know is that setup.py and pyproject.toml works with any Python build frontend, such as pip.

• If you need more control, the standard doc-comment at the start of setup.py explains in detail the various environment variables that can be used to control the build. If something is not clear, let me know and i'll clarify it.

  • Things changed quite a lot in release 1.23.0, because we've switched to using shared libraries. This is bound to effect how things are configured for Linux system installs.

• If someone is interested in using pipcl.py in a new project, pipcl.py has detailed documentation about how to use the Package class (which follows the general approach of distutils/setuptools), with a complete doctest example of bulding a Python extension using SWIG, and references to the relevant pypa specifications.

To comment on what @mjg has been writing earlier: The new way of building (even the a variant) seems to depend on different things than before. As noted in #2628 (comment) I have to provide other/ more libraries to the build and package those too. In this case extract, which appears to be bundled with mupdf and is not something we currently debundle because it is not used by anything else. Additionally, our libmupdf package currently only provides staticlibs (probably for the same reason as mentioned by @mjg).

Recent MuPDF has a new Extract sub-library, and so may require changes to how it is built. This is a separate issue from PyMuPDF.

It would be very helpful to first clear this up (and even add standardized integration, such as pkgconfig and/or cmake files to e.g. extract and mupdf) so that detection can happen generically across consumers and nothing has to be hardcoded by thousands of lines of custom Python code.

MuPDF builds are customised using environmental variables and/or make variables;, it does not use cmake or pkgconfig. You could suggest changes to the MuPDF team if you think they would be beneficial.