Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

numpy v1.26.0 with meson #297

Merged
merged 8 commits into from
Sep 17, 2023
Merged

numpy v1.26.0 with meson #297

merged 8 commits into from
Sep 17, 2023

Conversation

h-vetinari
Copy link
Member

Let's see if we can build with meson (esp. on windows...)

@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@h-vetinari
Copy link
Member Author

I based this off of the scipy recipe, and this went pretty far on unix:

Successfully built numpy-1.26.0b1-cp310-cp310-linux_x86_64.whl
+ pip install dist/numpy-1.26.0b1-cp310-cp310-linux_x86_64.whl
Processing ./dist/numpy-1.26.0b1-cp310-cp310-linux_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.26.0b1

However we then have:

INFO:conda_build.build:Packaging numpy
Packaging numpy
Packaging numpy-1.26.0b1-py310ha4c1d20_0
INFO:conda_build.build:Packaging numpy-1.26.0b1-py310ha4c1d20_0
No files or script found for output numpy
WARNING:conda_build.build:No files or script found for output numpy
number of files: 0

I think this might be because the default meson install sets a local --destdir:

[...]
[505/505] Linking target numpy/random/_generator.cpython-310-x86_64-linux-gnu.so
+ $PREFIX/bin/python $SRC_DIR/vendored-meson/meson/meson.py install --quiet --no-rebuild --destdir $SRC_DIR/.mesonpy-8wjmup75/install
[0/196] numpy/core/lib/libnpymath.a
[1/196] numpy/core/include/numpy/__multiarray_api.c
[...]

On windows, the same invocation as on unix fails:

(%BUILD_PREFIX%) %SRC_DIR%>%PREFIX%\python.exe -m build -w -n -x     -Cbuilddir=builddir     -Csetup-args=-Dblas=blas     -Csetup-args=-Dlapack=lapack     -Csetup-args=${MESON_ARGS// / -Csetup-args=} 
ERROR Source / does not appear to be a Python project: no pyproject.toml or setup.py

This is not a very comprehensive take yet, just a first shot. But it's getting late here, so this'll have to suffice for tonight.

@rgommers
Copy link
Contributor

I think this might be because the default meson install sets a local --destdir:

I would not expect that to be relevant, it's an implementation detail that's contained inside what meson-python is doing.

If the produced intermediate wheel is empty, pip would not report successfully installing it. If it's not empty, then I'm unclear why conda-build doesn't find any files. Perhaps easier to figure out from a local build than a CI log?

@h-vetinari
Copy link
Member Author

If it's not empty, then I'm unclear why conda-build doesn't find any files.

My suspicion is that it gets installed somewhere other than $PREFIX resp. SP_DIR.

Perhaps easier to figure out from a local build than a CI log?

Sure. As I said I haven't had much time for this yesterday, just started with opening the PR.

@h-vetinari
Copy link
Member Author

So the devdocs about building are not up-to-date, but that's being tracked, and at I also found numpy/numpy#24279. Not sure what's up with pkgconfig, our netlib blas/lapack builds do have the respective .pc files.

@h-vetinari
Copy link
Member Author

So for cross-compiling osx-arm64, we get

Checking for size of "long double" : 8 

../numpy/core/meson.build:461:2: ERROR: Problem encountered: Unknown long double format of size: 8

which seems to imply that the patterns used for detecting the long double flavour are perhaps not exhaustive.

@matoro, I see you authored numpy/numpy@f1cddd3, could it be that osx-arm64 has a different in-memory representation than what you've coded up for size 8?

@matoro
Copy link

matoro commented Aug 18, 2023

So for cross-compiling osx-arm64, we get

Checking for size of "long double" : 8 

../numpy/core/meson.build:461:2: ERROR: Problem encountered: Unknown long double format of size: 8

which seems to imply that the patterns used for detecting the long double flavour are perhaps not exhaustive.

@matoro, I see you authored numpy/numpy@f1cddd3, could it be that osx-arm64 has a different in-memory representation than what you've coded up for size 8?

Hey, I just tried to copy the python code 1-for-1. Does it somehow work on the setuptools implementation? Could you run this snippet and share the output?

#include <stdio.h>

const struct {
    const char before[16];
    const long double x;
    const char after[8];
} foo = {{'\0'}, -123456789.0, {'\0'}};

int main(void)
{
    for (size_t i = 0; i < sizeof(long double); i++) {
        fprintf(stdout, "%04o ", ((const unsigned char*)&foo.x)[i]);
    }
    fprintf(stdout, "\n");
    return 0;
}

@rgommers
Copy link
Contributor

The size of long double is 8 bytes on macOS arm64, and the format is IEEE_DOUBLE_LE in a native build. Since you're trying to cross-compile here, probably something is going wrong. The check does:

meson.get_compiler('c').run(...)

It should run under QEMU I think, and it likely doesn't here, or that is not working (can't tell from the CI log). You can backport numpy/numpy#24414 to bypass this check for now.

@h-vetinari
Copy link
Member Author

Thanks for the response @matoro and the reference @rgommers. This is looking good now!

Is the successful build with the beta enough from your POV (I'm guessing with a passing test suite we should be good normally?), or do you want to publish it to an rc-label to be able to test it against other packages?

@rgommers
Copy link
Contributor

Is the successful build with the beta enough from your POV (I'm guessing with a passing test suite we should be good normally?), or do you want to publish it to an rc-label to be able to test it against other packages?

This is fine I think for now; quite a few downstream packages are testing in CI themselves, because they already need it for Python 3.12 support. So I don't think we'll learn much more from a testing exercise in conda-forge now.

Thanks for verifying this beta though, great to see it working.

@matoro
Copy link

matoro commented Aug 18, 2023

This is a legit complaint though and it's my fault for breaking direct compatibility with cross-compiling. I don't think there is any way to make meson .run() work out of the box with cross-compiling. The original python code simply inspected the compiled object file rather than running it, so it worked. This needs to work natively without binfmt_misc set up, doing otherwise is a regression.

@mattip
Copy link

mattip commented Sep 10, 2023

I see the remaining CI failure is PyPy + windows, where meson --version is failing with an AttributeError: 'NoneType' object has no attribute 'encoding'.

@mattip
Copy link

mattip commented Sep 10, 2023

This seems to be some kind of pytest ineteraction? Using this works with no error:

conda create -n pypy3.9 pypy python=3.9
conda activate pypy3.9
mamba install meson
pypy -c "import subprocess; subprocess.check_call(["meson", "--version"])

@h-vetinari h-vetinari changed the title WIP: 1.26.0b1 numpy 1.26.0 Sep 16, 2023
@h-vetinari h-vetinari changed the title numpy 1.26.0 numpy v1.26.0 with meson Sep 16, 2023
@h-vetinari
Copy link
Member Author

This seems to be some kind of pytest interaction?

And/or meson interaction

  File "%PREFIX%\lib\site-packages\mesonbuild\mesonmain.py", line 230, in ensure_stdout_accepts_unicode
    if sys.stdout.encoding and not sys.stdout.encoding.upper().startswith('UTF-'):
AttributeError: 'NoneType' object has no attribute 'encoding'

@h-vetinari
Copy link
Member Author

Anyway, it looks like this'll have to get fixed on the meson-side. I don't think we should hold up 1.26 of win+pypy for now, it's already in high demand due to the impending 3.12.

@h-vetinari h-vetinari merged commit 05d6fd4 into conda-forge:main Sep 17, 2023
1 of 25 checks passed
@mbargull mbargull mentioned this pull request Nov 22, 2023
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants