Editable installs and PKG_CONFIG_PATH on OSX

[oscarbenjamin]

I am having trouble with editable installs of python-flint on OSX when using PKG_CONFIG_PATH. This follows #602 but it seemed better to break this out into a separate discussion. This is still related to the python-flint pull request flintlib/python-flint#129.

I am converting python-flint from setuptools to meson-python and as part of that I am using pkgconfig to locate its dependencies. Specifically python-flint depends on GMP, MPFR and Flint which are all C libraries. The current setuptools build uses environment variables like LD_LIBRARY_PATH to make everything work but using pkgconfig seems a lot better so I am trying to make that work for python-flint.

To reproduce what I show below you first need to do:

git clone https://github.com/oscarbenjamin/python-flint.git
cd python-flint
git checkout 4517a375d58d31adb4b6918382d9c0b9da789c9d
bin/build_dependencies_unix.sh

The commit hash is the current tip of the pr_meson_python branch that I am using for flintlib/python-flint#129. The bin/build_dependencies_unix.sh script takes 5-10 minutes depending on how fast the computer is. It downloads and builds GMP, MPFR and Flint, builds them with --prefix=$(pwd)/.local and installs all of them in the .local directory inside the python-flint directory.

At this point we are ready to build the python-flint extension modules and we can do e.g.:

$ pip install .
...
      Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 0.29.2
      Run-time dependency python found: YES 3.12
      Did not find CMake 'cmake'
      Found CMake: NO
      Run-time dependency gmp found: NO (tried pkgconfig and framework)

      ../meson.build:8:10: ERROR: Dependency "gmp" not found, tried pkgconfig and framework

The fails because we need to tell pkgconfig to find the dependencies in the .local directory. Instead we can do:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install .
$ python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

However the above does not work for an editable install on OSX:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --editable .
$ python -m flint.test -tq
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1262, in _find_spec
  File "/Users/enojb/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/_python_flint_editable_loader.py", line 271, in find_spec
    tree = self.rebuild()
           ^^^^^^^^^^^^^^
  File "/Users/enojb/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/_python_flint_editable_loader.py", line 312, in rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
  File "/Users/enojb/.pyenv/versions/3.12.2/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/ninja']' returned non-zero exit status 1.
$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig python -m flint.test
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1262, in _find_spec
  File "/Users/enojb/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/_python_flint_editable_loader.py", line 271, in find_spec
    tree = self.rebuild()
           ^^^^^^^^^^^^^^
  File "/Users/enojb/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/_python_flint_editable_loader.py", line 312, in rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
  File "/Users/enojb/.pyenv/versions/3.12.2/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/ninja']' returned non-zero exit status 1.

The above steps all work fine on Linux and there I can have an editable install and I can run python -m flint.test without needing to set any environment variables. Previously I think that the steps above were all working on OSX as well with meson-python. This problem has emerged since I switched to using pkgconfig and PKG_CONFIG_PATH as advised in #602.

Am I doing something wrong here?

[oscarbenjamin]

The above steps all work fine on Linux

This is not true. On Linux the LD_LIBRARY_PATH variable needs to be set at runtime when installing via pip:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install .
$ python -m flint.test -tq
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/oscar/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
    from .pyflint import *
  File "pyflint.pyx", line 1, in init flint.pyflint
  File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: libflint.so.19: cannot open shared object file: No such file or directory
$ LD_LIBRARY_PATH=$(pwd)/.local/lib python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!

However if I build wheels with build and install those then I don't need to set the environment variable:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig python -m build
$ pip install dist/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
$ python -m flint.test -tq  # no need to set LD_LIBRARY_PATH
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!

Somehow on Linux the wheels generated by pip wheel . or pip install . are different from those created by python -m build because the build wheels can be used without setting LD_LIBRARY_PATH. On OSX I don't see this difference between pip and build,

[oscarbenjamin]

Oh, wait now I'm seeing the same problem on Linux as well as OSX even though I was not seeing it before:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --editable .
$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig python -m flint.test
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1316, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1256, in _find_spec
  File "/home/oscar/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/_python_flint_editable_loader.py", line 271, in find_spec
    tree = self.rebuild()
           ^^^^^^^^^^^^^^
  File "/home/oscar/.pyenv/versions/tmp-python-flint/lib/python3.12/site-packages/_python_flint_editable_loader.py", line 312, in rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
  File "/home/oscar/.pyenv/versions/3.12.0/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/ninja']' returned non-zero exit status 1.

[dnicolodi]

I'm pretty sure that the ninja output tells you what is going wrong https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html#verbose-mode

[dnicolodi]

meson-python does exactly the same thing when it is invoked by pip or build, thus the problem is definitely somewhere else.

[oscarbenjamin]

Oh, no wait. I was forgetting --no-build-isolation when I wrote everything above. The editable install maybe works in all cases if that is used:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --editable .  --no-build-isolation
$ python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!

Now that I try to reproduce everything in fully fresh environments it seems that many of the errors I previously found are not applicable or are coming up different. I need to test this more...

[oscarbenjamin]

Okay, here is the original problem that I was investigating:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --no-build-isolation --editable . --no-clean
...
      ../../meson.build:8:10: ERROR: Dependency "gmp" not found, tried pkgconfig and framework

In the log I find:

Pkg-config binary missing from cross or native file, or env var undefined.
Trying a default Pkg-config fallback at pkg-config
Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 0.29.2
Determining dependency 'gmp' with pkg-config executable '/opt/homebrew/bin/pkg-config'
env[PKG_CONFIG_PATH]:
env[PKG_CONFIG]: /opt/homebrew/bin/pkg-config
-----------
Called: `/opt/homebrew/bin/pkg-config --modversion gmp` -> 1
stderr:
Package gmp was not found in the pkg-config search path.
Perhaps you should add the directory containing `gmp.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gmp' found

So somehow the PKG_CONFIG_PATH environment variable got unset by either pip, meson-python or meson. I'll try to make a simpler reproducer.

[QuLogic]

Make sure you clear the build directory before hand; Meson doesn't re-read environment variables after a build directory is created (otherwise your build could change if you forget to set it/set it differently on subsequent rebuilds).

[oscarbenjamin]

Meson doesn't re-read environment variables after a build directory is created

Thanks! This is exactly the information that I was missing.

Not understanding this point has probably been the main cause of my confusion about things seeming to work or not work in different situations.

I guess that when I call pip with pip install --no-build-isolation then if there is no build directory this will call meson setup which will store PKG_CONFIG_PATH in the build directory. If the build directory already exists then the previous value of PKG_CONFIG_PATH is already stored there and any new value is ignored.

If I always delete the build directory first then pip install --no-build-isolation --editable . seems to behave as I would expect and works both in OSX and Linux with the incremental rebuilds on import. Actually it is very nice that PKG_CONFIG_PATH is stored in the build directory so that I only need to set it once: now any python -m flint will rebuild correctly.

I am starting to see now why we probably need something like SciPy's dev.py to act as a front-end for setting things up...

[rgommers]

At this point I'd recommend https://github.com/scientific-python/spin/ as a reusable dev frontend, rather than a custom dev.py. spin is more or less equivalent to, and derived from, SciPy's dev.py.

Not understanding this point has probably been the main cause of my confusion about things seeming to work or not work in different situations.

I've seen this come up more; time to add a prominent .. note:: to the docs I think. It does have advantages (and will work like that with spin/dev.py too), e.g. it allows you to do:

CFLAGS='my-custom-flags' spin build
spin test  # no reconfigure
# open a separate terminal
spin test  # no still retain the custom CFLAGS rather than rebuilds everything, which is probably what you wanted

[oscarbenjamin]

At this point I'd recommend https://github.com/scientific-python/spin/ as a reusable dev frontend, rather than a custom dev.py

Thanks, I'll take a look at spin. I remember looking at it a year ago but I think I was just a bit overwhelmed trying to learn about all the different parts needed for a complete setup (meson, ninja, meson-python, mesonpy, pip, build, spin, ...).

One more question about storing environment variables. On OSX if I make wheels with pip wheel . or python -m build then I can install and use those wheels without setting DYLD_LIBRARY_PATH. On Linux I need to set LD_LIBRARY_PATH at runtime after installing the wheels or otherwise

ImportError: libflint.so.19: cannot open shared object file: No such file or directory

However when I do an editable install on Linux I can import and run the tests etc without setting LD_LIBRARY_PATH (either at build-time or run-time). So it seems that somehow in the case of the editable install the knowledge that we need this directory to be searchable for .so files is "stored" in the editable install.

Is the meson-python editable import hook adding the ./.local/lib directory to the search path somehow?

Is that stored in the build directory or in the editable install itself (maybe it doesn't matter since the two are tied together anyway)?

[eli-schwartz]

The python ecosystem usually tries to solve this with auditwheel/delocate anyway, I think.

[oscarbenjamin]

The python ecosystem usually tries to solve this with auditwheel/delocate anyway, I think.

Yes, that is how python-flint does it (and delvewheel for Windows).

It is somewhat unclear what the purpose is of creating a wheel that has not been "repaired". For me the purpose would just be to do pip install . but that does not currently work on Linux unless all non-Python dependencies are installed system-wide or you manipulate LD_LIBRARY_PATH at runtime.

[dnicolodi]

Python wheels are intended to be redistributable artifacts. This requires wheels to be built for a given platform, where the platform definition defines which libraries are available. This is what tools like cibuildwheel are for: making sure that wheels are built in a way to require only what is included in the platform definition. Extension modules linked to arbitrary shared libraries does not work in this scheme of things. What tools like auditwheel and delvewheel is to pack additional shared libraries required by the extensions modules inside a wheel into the wheel itself, making it redistributable accordingly to the definition above.

If you need to build a Python package for local use only, you don't need to build a wheel, thus the use case of extension modules in a wheel linking to shared libraries in arbitrary locations is not covered by meson-python. At the same time, because tools like auditwheel and delvewheel exist, meson-python does not bother checking that an extension module does not depend on shared libraries that are not part of the platform definition.

[eli-schwartz]

Wheels are required because pip requires a wheel in order to install because PyPA doesn't wish to be in the business of handling the complexities of generalist system software installation.

Hence wheels guarantee that installation is a pure extract+copy and the I/O of packing and unpacking a temporary wheel file is... hopefully insignificant.

[dnicolodi]

The script that builds these explicitly sets the path to the dependencies by passing --with-gmp=$(pwd)/.local when building MPFR and also --with-mpfr=$(pwd)/.local when building Flint. I don't know if that is what sets the RPATH entries or if it all just depends on me setting LD_LIBRARY_PATH.

It has been a long time since I worked with autotools, but IIRC the --with-foo flags usually result in an RPATH been set. I don't think setting LD_LIBRARY_PATH should be necessary, but you don't say when you set it and for which reason, so it is hard to tell.

So you have libflint having an RPATH set that allows it to find libgmp and libmpfr, and meson adding the location of libflint to the RPATH of the extension module. Therefore all works in an editable install that load the extension module directly from the build dir. When constructing the wheel, meson-python strips the RPATH, for the reasons explained by @eli-schwartz. Note how the RPATH added by meson is constructed having the build directory as root: /home/oscar/current/active/tmp/python-flint/build/cp312. meson-python only strips this type of RPATHs. In the case of a regular wheel build, the default is to build in an ephemeral build directory, thus, even if meson-python were to leave the RPATH in place, the RPATH would point to a non existing path (I think that, for security reasons, the dynamic linker insists that all elements of the RPATH must be traversable, thus the .. in the path do not short circuit).

[oscarbenjamin]

does `libflint.so itself depend on another shared library perhaps?

It links to some common system libraries and also to the GMP and MPFR that I build along with Flint:

$ ldd .local/lib/libflint.so
	linux-vdso.so.1 (0x00007ffeb845c000)
	libmpfr.so.6 => /home/oscar/current/active/python-flint/.local/lib/libmpfr.so.6 (0x00007f7a8d000000)
	libgmp.so.10 => /home/oscar/current/active/python-flint/.local/lib/libgmp.so.10 (0x00007f7a8e04b000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7a8d319000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7a8cc00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f7a8e0f2000)

After testing more carefully to remove all build directories and wheels, uninstall, purge the pip cache I find that python -m build works fine but pip wheel . does not. Somehow python -m build and pip wheel . create non-equivalent wheels.

This works:

PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig python -m build
pip install dist/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl 
python -m flint.test -tq

However pip wheel . fails at runtime (cannot open shared object file)

PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip wheel .
pip install python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
python -m flint.test -tq

Likewise pip install . fails at runtime (cannot open shared object file):

PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install .
python -m flint.test -tq

On the other hand pip's editable install works:

PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --no-build-isolation --editable .
python -m flint.test -tq

Here is pip wheel .:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip wheel .
Processing /home/oscar/current/active/python-flint
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: python-flint
  Building wheel for python-flint (pyproject.toml) ... done
  Created wheel for python-flint: filename=python_flint-0.6.0-cp312-cp312-linux_x86_64.whl size=2245204 sha256=8063c5e01e204acb2ac05c5613e716cc09304a63040596e9b92844e338ab5871
  Stored in directory: /home/oscar/.cache/pip/wheels/43/a9/f3/a40987b38ff4b4f29c92a95fea492d058f662d2a310271e407
Successfully built python-flint
$ pip install python_flint-0.6.0-cp312-cp312-linux_x86_64.whl 
Processing ./python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ python -m flint.test -tq
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/oscar/.pyenv/versions/python-flint-3.12/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
    from .pyflint import *
  File "pyflint.pyx", line 1, in init flint.pyflint
  File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: libflint.so.19: cannot open shared object file: No such file or directory
$ LD_LIBRARY_PATH=$(pwd)/.local/lib python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

Here is pip install .:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install .
Processing /home/oscar/current/active/python-flint
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: python-flint
  Building wheel for python-flint (pyproject.toml) ... done
  Created wheel for python-flint: filename=python_flint-0.6.0-cp312-cp312-linux_x86_64.whl size=2245204 sha256=ccafa952b6b9c275f3d02c1425c4191aa2bbc40e34fd3c517013938a7bb38fb0
  Stored in directory: /home/oscar/.cache/pip/wheels/43/a9/f3/a40987b38ff4b4f29c92a95fea492d058f662d2a310271e407
Successfully built python-flint
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ python -m flint.test -tq
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/oscar/.pyenv/versions/python-flint-3.12/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
    from .pyflint import *
  File "pyflint.pyx", line 1, in init flint.pyflint
  File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: libflint.so.19: cannot open shared object file: No such file or directory
$ LD_LIBRARY_PATH=$(pwd)/.local/lib python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

This is python -m build:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig python -m build
<snip longer output>
$ pip install dist/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl 
Processing ./dist/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

This is an editable build:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip install --no-build-isolation --editable .
Obtaining file:///home/oscar/current/active/python-flint
  Checking if build backend supports build_editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: python-flint
  Building editable for python-flint (pyproject.toml) ... done
  Created wheel for python-flint: filename=python_flint-0.6.0-cp312-cp312-linux_x86_64.whl size=7451 sha256=fc90c9b87434a0a73d1c4085d30b32ce75ddd72c51e53c394a2c3a5d458eacc5
  Stored in directory: /tmp/pip-ephem-wheel-cache-vwd9jqkj/wheels/43/a9/f3/a40987b38ff4b4f29c92a95fea492d058f662d2a310271e407
Successfully built python-flint
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

[oscarbenjamin]

Somehow python -m build and pip wheel . create non-equivalent wheels.

I tried making a standalone script to reproduce this from scratch but couldn't:

#!/bin/bash

git clone https://github.com/oscarbenjamin/python-flint.git
cd python-flint
git checkout 4517a375d58d31adb4b6918382d9c0b9da789c9d
bin/build_dependencies_unix.sh

pip install build
PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig python -m build --wheel --outdir build_wheel
PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig pip wheel --wheel-dir pip_wheel .

With that script I get wheels from python -m build and pip wheel . that have the exact same filesize but are not byte for byte identical:

$ ls -l build_wheel/ pip_wheel/
build_wheel/:
total 2200
-rw-rw-r-- 1 oscar oscar 2245342 Mar 24 18:52 python_flint-0.6.0-cp312-cp312-linux_x86_64.whl

pip_wheel/:
total 2200
-rw-rw-r-- 1 oscar oscar 2245342 Mar 24 18:54 python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
$ diff -qs build_wheel/ pip_wheel/
Files build_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl and pip_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl differ

Both wheels seem to behave the same in respect of needing to use LD_LIBRARY_PATH:

$ pip install build_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl 
Processing ./build_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ python -m flint.test -tq
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/oscar/.pyenv/versions/3.12.0/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
    from .pyflint import *
  File "pyflint.pyx", line 1, in init flint.pyflint
  File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: libflint.so.19: cannot open shared object file: No such file or directory
$ LD_LIBRARY_PATH=$(pwd)/.local/lib python -m flint.test -tq
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------
$ pip uninstall python-flint
Found existing installation: python-flint 0.6.0
Uninstalling python-flint-0.6.0:
  Would remove:
    /home/oscar/.pyenv/versions/3.12.0/lib/python3.12/site-packages/flint/*
    /home/oscar/.pyenv/versions/3.12.0/lib/python3.12/site-packages/python_flint-0.6.0.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled python-flint-0.6.0
$ pip install pip_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl 
Processing ./pip_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
$ python -m flint.test -tq
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/home/oscar/.pyenv/versions/3.12.0/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
    from .pyflint import *
  File "pyflint.pyx", line 1, in init flint.pyflint
  File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: libflint.so.19: cannot open shared object file: No such file or directory
$ pip install pip_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl 
Processing ./pip_wheel/python_flint-0.6.0-cp312-cp312-linux_x86_64.whl
python-flint is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

So I can't reproduce the effect I saw before where python -m build and pip wheel . give non-equivalent wheels. The difference is just that in editable mode LD_LIBRARY_PATH is not needed where as when installing from the built wheels it is necessary to set LD_LIBRARY_PATH (on Linux bot not on OSX).

[dnicolodi]

With that script I get wheels from python -m build and pip wheel . that have the exact same filesize but are not byte for byte identical

This is probably due to some file metadata in the zip being different. File modification times, for example, of something of that sort. I think that building twice in a row with the same frontend does not result in a byte-for-byte identical wheel either. I think this is a bug, because we strive to make the build reproducible. diffoscope could be useful to see in what the wheels actually differ.

The difference is just that in editable mode LD_LIBRARY_PATH is not needed where as when installing from the built wheels it is necessary to set LD_LIBRARY_PATH (on Linux bot not on OSX).

This is expected. See the explanation about RPATH handling a few messages above.

[oscarbenjamin]

diffoscope could be useful to see in what the wheels actually differ.

According to diffoscope only a few bytes were different but I'm not sure exactly how to interpret the hd diff output. When extracting the wheels and comparing the extracted trees there are no differences:

$ diff -qs pip_wheel/flint/ build_wheel/flint/
Common subdirectories: pip_wheel/flint/flint_base and build_wheel/flint/flint_base
Common subdirectories: pip_wheel/flint/functions and build_wheel/flint/functions
Files pip_wheel/flint/__init__.py and build_wheel/flint/__init__.py are identical
Files pip_wheel/flint/pyflint.cpython-312-x86_64-linux-gnu.so and build_wheel/flint/pyflint.cpython-312-x86_64-linux-gnu.so are identical
Common subdirectories: pip_wheel/flint/test and build_wheel/flint/test
Common subdirectories: pip_wheel/flint/types and build_wheel/flint/types
Common subdirectories: pip_wheel/flint/utils and build_wheel/flint/utils
$ diff -qs pip_wheel/python_flint-0.6.0.dist-info/ build_wheel/python_flint-0.6.0.dist-info/
Files pip_wheel/python_flint-0.6.0.dist-info/METADATA and build_wheel/python_flint-0.6.0.dist-info/METADATA are identical
Files pip_wheel/python_flint-0.6.0.dist-info/RECORD and build_wheel/python_flint-0.6.0.dist-info/RECORD are identical
Files pip_wheel/python_flint-0.6.0.dist-info/WHEEL and build_wheel/python_flint-0.6.0.dist-info/WHEEL are identical

The differences must just be file metadata as you suggest.

[oscarbenjamin]

When building a regular package, the extra RPATH entries are stripped.

What is the reason for stripping them?

The simplest reason is because some of those rpath entries could look like:

/home/jane/code/mycoolproject/builddir/lib:/home/jane/code/mycoolproject/builddir/util

and it's a security vulnerability to ship those rpaths to production systems

Does this consideration not apply equally to Linux and OSX?

This is what I get after extracting a wheel generate by python -m build on OSX:

$ otool -L dist/flint/types/fmpz.cpython-312-darwin.so
dist/flint/types/fmpz.cpython-312-darwin.so:
	/Users/enojb/work/dev/python-flint/.local/lib/libflint.19.0.dylib (compatibility version 19.0.0, current version 19.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

I can install this wheel and I can use it without needing to set DYLD_LIBARY_PATH. On Linux though it seems that the RPATH entries are stripped so I get a wheel that I can install but I can't import the modules unless I set LD_LIBRARY_PATH.

[eli-schwartz]

https://github.com/mesonbuild/meson/blob/master/mesonbuild/scripts/depfixer.py

The handling for macOS and Linux is different because one uses install_name_tool and the other uses an elf editor. I'm afraid I don't really have personal experience with the macOS side of things.

[oscarbenjamin]

Trying this on a different Linux machine I find that the RPATH entries are not stripped and so the behaviour is similar to macos.

Maybe I've changed something that has fixed this. I'll have to try again on the original Linux machine...