Python purelib/platlib hardcoding breaks prefix detection with meson

[stewartadam]

brew gist-logs <formula> link OR brew config AND brew doctor output

https://gist.github.com/stewartadam/64f2b9a51e6e847c380a134823e000d7

Verification

• My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

Install python module to brew-provided python using meson.

brew patches python's sysconfig with hardcoded, absolute paths to /opt/homebrew/... for purelib and platlib. This breaks prefix detection for installers like 'meson', rendering them unable to install python modules easily.

This can be verified using python3 -c "import pprint;import sysconfig; pprint.pprint(sysconfig.get_paths(vars={'base': '', 'platbase': '', 'installed_base': ''}, expand=False))".

Fedora 37 (3.11.1)

{'data': '{base}',
 'include': '{installed_base}/include/python{py_version_short}{abiflags}',
 'platinclude': '{installed_platbase}/include/python{py_version_short}{abiflags}',
 'platlib': '{platbase}/{platlibdir}/python{py_version_short}/site-packages',
 'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
 'purelib': '{base}/lib/python{py_version_short}/site-packages',
 'scripts': '{base}/bin',
 'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}'}

brew python3.10

{'data': '/opt/homebrew',
 'include': '{installed_base}/include/python{py_version_short}{abiflags}',
 'platinclude': '{installed_platbase}/include/python{py_version_short}{abiflags}',
 'platlib': '/opt/homebrew/{platlibdir}/python{py_version_short}/site-packages',
 'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
 'purelib': '/opt/homebrew/lib/python{py_version_short}/site-packages',
 'scripts': '/opt/homebrew/bin',
 'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}'}

What happened (include all command output)?

The Meson build system
Version: 1.0.0
Source dir: /Users/stewartadam/repro
Build dir: /Users/stewartadam/repro/_build
Build type: native build
Project name: foo
Project version: 1.0.0
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python3 found: YES (/opt/homebrew/bin/python3)
Message: prefix /opt/homebrew
Message: bindir bin
Message: pylib /opt/homebrew/opt/homebrew/lib/python3.10/site-packages/
Message: purelib /opt/homebrew/lib/python3.10/site-packages
Message: stdlib /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10
Build targets in project: 0

Found ninja-1.11.1 at /opt/homebrew/bin/ninja

Setting prefix=/ isn't helpful because then it gets the python paths right, but tries to install to read-only /bin, /share, etc.

What did you expect to happen?

Lack of a duplicate /opt/homebrew/opt/homebrew prefix.

Since the constructed value {platbase}/{platlibdir}/python{py_version_short}/site-packages points to a symlink of /opt/homebrew/lib/python3.10/site-packages anyways, please consider using it instead to permit build systems to construct paths using a user-specified prefix.

This will permit package managers to obtain the path components under the brew python installation directory that they should apply along with the brew prefix.

Step-by-step reproduction instructions (by running brew commands)

brew install meson python3
mkdir repro
cd repro
cat << EOF > meson.build
project('foo',
    version: '1.0.0',
    meson_version: '>= 0.59.0',
)
python = import('python').find_installation('python3')
pylib = python.get_install_dir()
message('prefix', get_option('prefix'))
message('bindir', get_option('bindir'))
message('pylib', pylib)
message('purelib', python.get_path('purelib'))
message('stdlib', python.get_path('stdlib'))
EOF
meson setup _build

[Bo98]

Given there's nothing technically wrong with our configuration and Fedora's configuration - which you're comparing our configuration to - is equally affected by this issue, this sounds like a Meson bug?

Since the constructed value {platbase}/{platlibdir}/python{py_version_short}/site-packages points to a symlink of /opt/homebrew/lib/python3.10/site-packages anyways

The problem is some packages bake this path into their installs and it is not a stable path.

This will permit package managers to obtain the path components under the brew python installation directory that they should apply along with the brew prefix.

purelib is not the correct way to retrieve base Python installation files - stdlib is. Unless I'm misunderstanding what you're saying.

please consider using it instead to permit build systems to construct paths using a user-specified prefix.

Like Fedora, we use a system that makes our configuration not take effect in cases the prefix changes (sys.prefix != sys.base_prefix), such as virtual environments.

Python's sysconfig system is still extremely restrictive. There is no real system to swap out what {base} and {platbase} refer to - there has been some chat in the Python issue tracker about this. Fedora's also affected by this. Supplying --prefix to setuptools is not supported as a result. pip has a workaround for this limitation and so is not affected.

Actually reading this again it sounds like Meson might be affected by the same shortcoming?

[stewartadam]

The problem is some packages bake this path into their installs and it is not a stable path.

Ah, that makes sense (but is unfortunate).

This will permit package managers to obtain the path components under the brew python installation directory that they should apply along with the brew prefix.

purelib is not the correct way to retrieve base Python installation files - stdlib is. Unless I'm misunderstanding what you're saying.

Sorry I mean to say 'build systems', not 'package managers'. I was trying to convey that build systems often have a destination directory (configured via some prefix) and install everything under there. Since there isn't really a good mechanism upstream in Python for this, its best guess would be to check sysconfig with {'base': '', 'platbase': '', 'installed_base': ''} but absolute paths break that method of getting base/unprefixed paths for stdlib and pathlib.

What I was hoping for was --prefix=/opt/homebrew would result in pylib of /opt/homebrew/lib/python3.10/site-packages/ and --prefix=~/.local would instead result in ~/.local/lib/python3.10/site-packages/.

Because the path is absolute and base, etc cannot be collapsed to empty string - we get /opt/homebrew/opt/homebrew/<..> and ~/.local/opt/homebrew/<...> instead.

Actually reading this again it sounds like Meson might be affected by the same shortcoming?

Yeah, that's it - it seems like this is a gap in Python itself and everyone's is trying to work with what we have, but the net result is things don't interplay very well. I was hoping a change here could easily net positive results for meson and build systems using a similar technique, but the rationale you mentioned for preserving the absolute path makes sense.

I don't think meson is doing anything wrong, it makes sense from its perspective to disallow writing files outside the configured prefix/destdir but on the other hand, it makes constructing the destination path difficult when python installations can use paths without {base} and the likes.

[eli-schwartz]

it seems like this is a gap in Python itself and everyone's is trying to work with what we have, but the net result is things don't interplay very well

Absolutely.

This is awkward handling around an awkward ecosystem, if there's anything really to "blame" for the situation it's the fact that effectively undocumented interfaces have changed inside their undocumented guts and change is fragile.

It doesn't help that python upstream doesn't have much of a usability story for /usr vs. /usr/local split and it's actually not exactly easy to reverse-engineer the venv vs. not-venv split either.

Like Fedora, we use a system that makes our configuration not take effect in cases the prefix changes (sys.prefix != sys.base_prefix), such as virtual environments.

I'm not 100% sure what this means.

In at least Fedora's case, the conflict was caused by the fact that Fedora patched sysconfig to say that when prefix is /usr, python files should still be installed to /usr/local, which needless to say is actually a Fedora bug because it's not like Fedora patches autotools to make --prefix=/usr install libraries to /usr/local/lib except inside of a Fedora buildroot, that would be insane. 🙄

But no, pip is special and magical and everyone has to design the entirety of the python packaging ecosystem around the assumption that pip, and only pip, is ever used. :D

As far as virtualenvs go, I'm not sure what the problem is? Usually you do want virtualenvs to install into a different location from the global install path. That is, for example, why Fedora only patches "/local" into the scheme when it detects that you're not running in a virtualenv and you're not running inside of a Fedora rpm buildroot. Inside of a virtualenv, python's official scheme replacements for platbase and base are relative to the virtual environment, since sys.prefix is the virtual environment prefix (and the scheme detects the stdlib using base_prefix).

This doesn't invalidate the concept of using those replacements -- it's just that sometimes distributors want the layout to have "literally nothing" to do with how python was installed, simply to stop pip from installing to it. And sysconfig doesn't have an option like layout_prefix='/usr' to say "yes I actually would like to see the install scheme that assumes I am going to install to /usr".

[tristan957]

Fedora fixed this in Fedora 37, fyi. I opened up a bug report in Fedora 36, which was subsequently fixed.

https://bugzilla.redhat.com/show_bug.cgi?id=2097183

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tristan957]

Leave it to stalebot to close valid issues in the name of project management.

[carlocab]

It's not clear what the right fix is, nor does it look like we have the bandwidth to work one out. We'll review a pull request for this, however.

As a workaround, it may suffice to pass --install-lib to setup.py, or -Dpython.{pure,plat}libdir to meson.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tristan957]

It's not clear what the right fix is, nor does it look like we have the bandwidth to work one out. We'll review a pull request for this, however

Sure. No one is asking for a fix this second or any time specifically. It just does users no good at all to have a stale bot closing valid issues. When users search for a bug report, GitHub hides closed issues by default, which will lead to a duplicate bug report.

[SMillerDev]

It does maintainers no good to have a never ending list of issues with no indication if they still occur. And when those are the options we optimise for the thing we have less of.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.