Build failures using meson-python 0.11.0 with cibuildwheel

[dnicolodi]

On Windows, building for win32 fails with

Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\cibw-run-dbnhe2vj\cp37-win32\build\venv\Scripts\python.exe)
Need python for 64-bit, but found 32-bit
        
..\..\meson.build:24:3: ERROR: Python dependency not found

it may be a cibuildwheel issue or a Meson issue. Full log here https://github.com/dnicolodi/python-siphash24/actions/runs/3517527064/jobs/5895402010

On macOS, building succeeds but test fail. The test pass when the package is not built with cibuildwheel. No idea what is going on as it seems that the test process segfaults. Maybe a linking issue. Full log here https://github.com/dnicolodi/python-siphash24/actions/runs/3517527064/jobs/5895402097

On Linux, linking issue:

FAILED: siphash24.cpython-37m-x86_64-linux-gnu.so
        cc  -o siphash24.cpython-37m-x86_64-linux-gnu.so siphash24.cpython-37m-x86_64-linux-gnu.so.p/meson-generated_siphash24.pyx.c.o -Wl,--as-needed -Wl,--allow-shlib-undefined -shared -fPIC -Wl,--start-group subprojects/libcsiphash-1/src/libcsiphash-1.a -L/opt/_internal/cpython-3.7.13/lib -lpython3.7m -Wl,--end-group
        /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -lpython3.7m

it may be Python 3.7 specific. Again not seen when building without cibuildwheel, maybe related to the setup used in the manylinux container. Full log here https://github.com/dnicolodi/python-siphash24/actions/runs/3517527064/jobs/5895401917

EDIT: a summary of the actual issue is in #222 (comment), and a Meson issue is linked from #222 (comment). There is also useful discussion on cross-compiling in the comments below.

[dnicolodi]

@rgommers SciPy uses cibuildwheel. Did you bump into any of these issues? Don't you build for these platforms?

[henryiii]

Side comment: When Meson fails, we should raise SystemExit (like setuptools usually does), not throw an exception. Pip/build report a mess when there's an uncaught exception from the build hooks, while SystemExit is handled gracefully without nearly as many extra reports about missing build hooks and changed exceptions. The actual failure is printed by Meson.

[henryiii]

Is this an extension or an app? You should not be linking to Python on macOS/linux if it's an extension. The libs are intentionally deleted from the manylinux images to stop users from trying to link to them.

That Windows failure is supposed to be building a 32-bit wheel, so guessing it's either meson or meson-python not respecting the attempt to build for 32-bit? It does find the 32-bit Python then complains when it's 32-bit (which is what it is supposed to be).

[dnicolodi]

Is this an extension or an app? You should not be linking to Python on macOS/linux if it's an extension. The libs are intentionally deleted from the manylinux images to stop users from trying to link to them.

It is an extension. Meson takes care of the linking, when needed. I don't do it explicitly. Interestingly, it is a problem only for Python 3.7 in cibuildwheel. I've no idea what's going on.

That Windows failure is supposed to be building a 32-bit wheel, so guessing it's either meson or meson-python not respecting the attempt to build for 32-bit? It does find the 32-bit Python then complains when it's 32-bit (which is what it is supposed to be).

The Windows issue I think is due to cibuildwheel not setting up things for Meson to find the MSVC compiler. I'm trying to setup the correct environment variables from outside to see if this fixes cibuildwheel.

[dnicolodi]

The Windows issue I think is due to cibuildwheel not setting up things for Meson to find the MSVC compiler. I'm trying to setup the correct environment variables from outside to see if this fixes cibuildwheel.

The one solution I've tried https://github.com/bus1/cabuild/tree/main/action/msdevshell does not work:

        The Meson build system
        Version: 0.64.0
        Source dir: D:\a\python-siphash24\python-siphash24
        Build dir: D:\a\python-siphash24\python-siphash24\.mesonpy-lnm4qgxr\build
        Build type: native build
        Project name: python-siphash24
        Project version: undefined
        
        ..\..\meson.build:1:0: ERROR: Found GNU link.exe instead of MSVC link.exe in C:\Program Files\Git\usr\bin\link.EXE.
        This link.exe is not a linker.
        You may need to reorder entries to your %PATH% variable to resolve this.

[henryiii]

Are you using the action? There's a report that by using the bash shell in the aciton this breaks Meson's discovery. I think we can fix it by using the native shell and splitting the action up (though, why does this break meson? CMake and setuptools are just fine as it is).

Meson should not be putting in -lpython3.7(m), that's a bug when building something to be distributable. You can't link to Python's libs. That will cause segfaults on macOS and libraries not found on manylinux. It will work when building locally, since you are using the same patch release of Python that you built with, so it won't segfault (unless you upgrade Python to a new patch release without rebuilding, or sending it to someone else, etc.)

[henryiii]

This one: pypa/cibuildwheel#1338

[henryiii]

Also, ilammy/msvc-dev-cmd is what I usually use.

[dnicolodi]

Meson does the same as thing distutils does to decide when to link to libpython. It actually calls into distutils, see https://github.com/mesonbuild/meson/blob/3ae89a7150264ae5a112975af1377ee1c04cb994/mesonbuild/modules/python.py#L366-L370 I'm surprised that distutils is broken in this way.

[dnicolodi]

Also, ilammy/msvc-dev-cmd is what I usually use.

In combination with the cibuildwheel action? How do you set the architecture when you use cibuildwheel to compile both amd64 and x86?

[henryiii]

That rather scares me - is Meson being tested on Python 3.12a2? Distutils has been removed there. I thought the point of moving to Meson was to be ready for distutils' removal?

If that's the built-in distutils, it wouldn't horribly surprise me, as it's usually filtered by setuptools, and would explain why it would be Python version specific. https://github.com/pypa/setuptools/blob/59ee4980a0f49ea610e26a1aca104334ae03d140/setuptools/_distutils/command/build_ext.py#L736-L789

Hmm, see https://github.com/pypa/setuptools/blob/59ee4980a0f49ea610e26a1aca104334ae03d140/setuptools/_distutils/command/py37compat.py#L6. See pypa/distutils#9.

[henryiii]

I generally don't mix the two because both setuptools and CMake are smart enough to work without ilammy/msvc-dev-cmd. The first thing I'd recommend trying is swapping the action for run: pipx run cibuildwheel==2.9.0 and see if that fixes it. If it does, I'll see if we can split the action up into Unix and Windows steps. (Also, the main reason to use the action is to use dependabot to keep it up to date, which I'm guessing you aren't doing since you are using 2.9.0 instead of 2.11.2).

Otherwise, you can split the job up into 32 bit and 64 bit steps. It's fine to run msvc-dev-cmd twice. (In fact, I vaguely think I have done this before for something).

[dnicolodi]

The fact that distutils is going away is know, but there are reasons why Meson depends on it. @eli-schwartz is the one that has the whole picture and working on a solution. You may want to discuss the details with him.

[dnicolodi]

cibuildwheel uses bash in GitHub Actions and this puts the GNU compilers on the $PATH making a mess for Meson. The solutions for having MSVC compilers on the path do not work for this reason. It seems that cibuildwheel and Meson simply do not mix. One way around is to use the Mingw compilers adding the required configuration to meson.build, but this does not allow to build win32 wheels, I think.

[henryiii]

Did you try my pipx suggestion? That will avoid bash on Windows. (And that's basically the solution we'd propose in cibuildwheel if it works)