python module: Respect PATH when python is not given in machine file

[chewi]

We should only fall back to the Python interpreter running Meson itself if python3 is not found in the PATH.

See Gentoo bug #912051.

[chewi]

Arrgh. I tried to fix the failing Arch/PyPy tests by amending PATH, as they relied on the old behaviour, but this broke other Python-based tools like g-ir-scanner. Gentoo is smart as it uses its python-exec wrapper or otherwise adjusts the shebang to prevent the PATH from screwing things up like this. Unfortunately, Arch is not that smart. Maybe I'll have to fix the tests with a toolchain file instead, but I'm not sure how easy this will be yet.

[Volker-Weissmann]

You should squash both commits into one commit:

https://mesonbuild.com/Contributing.html#strategy-for-merging-pull-requests-to-trunk

[Volker-Weissmann]

You need to update the documentation on how python is found:

https://github.com/mesonbuild/meson/blob/master/docs/markdown/Python-module.md

[Volker-Weissmann]

LGTM

[chewi]

CI is killing me and I'm close to giving up. As far as I can tell, the Arch/PyPy job normally builds the boost test case against CPython because meson.build explicitly asks for python3. By pointing python in the native file to PyPy, this forces it to use that instead, and that somehow causes the detection to fail. I don't know whether this test case is really supposed to use CPython or not. Maybe it doesn't make sense for PyPy and we should skip it? Or are you explicitly testing that it doesn't choose PyPy here? We don't have a way to change the native file for just one test case. Advise appreciated.

[konsolebox]

Maybe keep old behavior instead but add option to make the function follow PATH first? This at least will avoid surprises in existing setups. I'm not sure however how it will help Gentoo's case.

[eli-schwartz]

I talked about this a bit on the gentoo ticket too.

This (the find_installation code for determining which path to query as name_or_path) is some pretty old code and it's been in the back of my mind to fix it for a while. What actually happens is a bit complex:

• find_installation() with no arguments will use meson's own sys.executable
• find_installation('python3') will use whichever python3 executable is on PATH
• find_installation(get_option('python_bin')) ditto except selected by the project option -Dpython_bin=whatever
• setting the binary path for python = 'whatever' in a machine file overrides detection

I agree that ideally without arguments we would first look up a python3 from PATH. Also if python3 is explicitly specified but cannot be found (like on crazy FreeBSD systems where they refuse to install a python3 symlink to python3.9 or whatever is current), we should fall back to sys.executable. And the machine file lookup should prefer checking for a versioned python entry before checking for an unversioned entry.

[chewi]

@konsolebox An option seems like an ugly compromise. I'd rather just using a machine file under Gentoo. The only reason I'm proposing this fix is because I feel quite strongly that the existing behaviour makes no sense. The fact that Meson is written in Python is an implementation detail, never mind the Python version it is running under. Muon is a Meson implementation written in C, so what would you expect that to do here?

The output of the failing boost test case was confusing me, so I reproduced the failure with PyPy locally and unearthed another bug. If python was pointed at Python 3 in a machine file, and that Python failed to be recognised as python2 then a subsequent search for python3 would fail because the previous failure was cached. I have attempted to fix this, but Eli has already provided feedback that needs addressing.

I have also had to exclude the boost test case on Arch/PyPy because it simply cannot work against PyPy. It was previously using CPython, despite Meson itself running under PyPy, but the machine file has changed that.

[chewi]

Arch/PyPy has started passing at last. Not sure why the Windows targets are failing now. 😕

[chewi]

I've just seen someone else's branch fail on the Windows targets in exactly the same way. Probably not my fault then?

[tristan957]

Known failures.

[chewi]

CI has stopped playing up. In case it wasn't clear, I have addressed @eli-schwartz's feedback above. Please merge.