pyproject.toml

[totaam]

Already added in 1f47f78

More should be added to it: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/

[totaam]

It triggers some new warnings:

The following seems to be defined outside of `pyproject.toml`:

        `urls = {'Documentation': 'https://github.com/Xpra-org/xpra/tree/master/docs',
                     'Funding': 'https://github.com/sponsors/totaam',
                     'Source': 'https://github.com/Xpra-org/xpra'}`
...

[totaam]

Good enough for now. See also #4086

What's not solved is how setuptools is desperately trying to force us not to install data files outside the python directory, despite the fact that this is where the files need to go and every project is facing the same conundrum.

[eli-schwartz]

What's not solved is how setuptools is desperately trying to force us not to install data files outside the python directory, despite the fact that this is where the files need to go and every project is facing the same conundrum.

This isn't a setuptools problem.

The Python Packaging Authority ("not an authority") has authoritatively stated that:

• pip and similar ecosystem tools shall only install python software using python wheels
• python wheels are intended for importable libraries, not applications, unless your application only installs:
  • importable library code
  • CLI script entrypoints to some location in $PATH

The wheel format does not permit you to install files to /etc, so you can simply quit that line of thought immediately. It is not happening, end of story.

You can continue using python setup.py install --root="$DESTDIR" for as long as setuptools chooses to leave it in "deprecated" mode instead of removing it entirely, but pip install won't handle that anyway.

Ultimately, PyPA wants to "get out of the game". Back in the day, setup.py was the python ecosystem's answer to cross-platform Makefiles that:

• run on Windows
• have native support for detecting the correct installation directory for *.py files
• provide glue logic for compiling dlopen'able shared libraries that correctly build and link to the Python headers and libpython

... but you could use that however you liked, it was "just a toolkit".

They've since decided that supporting arbitrary application packaging is hard work and out of scope for the PyPI.org ecosystem of distributable wheels, and wish to refocus their efforts on doing one thing and doing it well, and removing support for users who want to do two things.

[eli-schwartz]

You may be interested in porting to meson (disclaimer: I am a meson developer) since it:

• supports actual application packaging à la autotools
• unlike autotools is actually fast and ergonomic
• supports python very well (it can be equally used for actual application packaging, or for distributing library wheels to PyPI via meson-python as a setuptools replacement in a pyproject.toml build-system.build-backend)

[totaam]

@eli-schwartz thanks, I didn't think of switching to meson. I believe that all the platforms we target have it installed.
That said, I am not looking forward to converting 2450 lines of setup.py.
Also, the few times that I had to figure out how to change things with meson, I couldn't get a clear picture of where to look and how to go about things - not that I would prefer autotools!

.. and removing support for users who want to do two things.

And that is what is so infuriating.
By all means, ask projects to move in a different direction and ideally provide a clear migration path.
But breaking other developers' projects, and on purpose, that's quite ridiculous.

[eli-schwartz]

That said, I am not looking forward to converting 2450 lines of setup.py.

Yeah. :( Porting between build systems is hard work, especially the "check that it is feature compatible" stage.

That said, given setup.py install is deprecated with intent to remove at some point, so really I believe it basically has to be done at some point, and the earlier the better since it gives more time to test a replacement before setup.py install disappears.

There's some good news to go with the bad news, though: a decent amount of what's going on in setup.py is manually implemented things that meson has native functionality for.

Also, the few times that I had to figure out how to change things with meson, I couldn't get a clear picture of where to look and how to go about things - not that I would prefer autotools!

I would be happy to provide advice, please feel free to @ me if you have questions.

[totaam]

@eli-schwartz you may be regretting this pretty quickly!
Where would I even begin to figure out why the dylibs end up with the wrong path:
Xpra-org/gtk-osx-build#23
Xpra-org/gtk-osx-build#19 (comment)

[eli-schwartz]

So the problem is that you are getting @rpath/libfoo.dylib entries and you'd really rather like to get absolute paths?

[totaam]

@eli-schwartz yes, we need an absolute path without any @rpath in it.
That's what every other library does (and we have literally hundreds of them) without requiring anything specific, only these 2 seem to cause problems.
And more importantly, how do I figure out the solution in meson? (so I don't have to get stuck or ask for help next time I encounter such an issue)

[eli-schwartz]

Meson has (optional) settings per target for build time and install time RPATH. It also generates internal build time RPATH entries per target for each dependency library containing the absolute path, which is necessary to ensure uninstalled running of your programs, whether that is for:

• meson devenv
• running tests
• build time tools that get run to produce additional sources
• just cd'ing in and running something (not guaranteed to work if you have dedicated $PATH requirements or project specific environment variables which e.g. point at uninstalled data/resources. This is why devenv exists)

At install time, a depfixer is run to rewrite rpath to something suitable for installation. On macOS, naturally, this runs install_name_tool. So it's easy to grep for in the codebase. Anyways, it is here:

https://github.com/mesonbuild/meson/blob/master/mesonbuild/scripts/depfixer.py

The fix_darwin function (as opposed to fix_elf) looks like this excerpt:

        if new_rpath:
            args += ['-add_rpath', new_rpath]
        # Rewrite -install_name @rpath/libfoo.dylib to /path/to/libfoo.dylib
        if fname.endswith('dylib'):
            args += ['-id', final_path]
        if install_name_mappings:
            for old, new in install_name_mappings.items():
                args += ['-change', old, new]
        if args:
            subprocess.check_call(['install_name_tool', fname] + args,
                                  stdout=subprocess.DEVNULL,
                                  stderr=subprocess.DEVNULL)

The comment particularly notes that we get rid of those @rpath strings and replace it with absolute paths.

This works fine for any libraries built by meson. At https://wrapdb.mesonbuild.com you can see that someone has contributed a "meson wrap" for libjpeg-turbo, which allows you to do this:

mkdir -p subprojects
meson wrap install libjpeg-turbo

and install a tiny INI formatted *.wrap file into your source tree; when meson sees dependency('libjpeg') it will first look using pkg-config to find that dependency, and if it can't find it, it will use the url and checksum in the INI file to download the dependency and build a private copy using an embedded meson.build file patched into the upstream source tarball by the wrap file.

(This is the default search pattern. There's a configure option for people who would rather a different policy, such as "always use vendored subprojects" or "our policy is that vendoring is banned, just fail if it isn't available", or "anything is fine as long as no network connections are made".)

For libavif, since no one wrote a meson port for the WrapDB, it is "probably" possible to use meson's cmake module to configure libavif and parse the cmake debug tracing API, retrieve build definitions, and convert it to a meson AST then build it with meson. In case you're wondering, yes, that is exactly as hacky as it sounds. :D However, nontrivial effort has been put into making it work, and for the most part that has succeeded.