py-numpy: add v2.0.0

[adamjstewart]

https://github.com/numpy/numpy/releases/tag/v2.0.0

Excited to see how many packages break. We could pin the default version to v1, but I'd rather fix any dependency bugs that pop up instead.

[adamjstewart]

Extremely helpful! numpy/numpy#26191

[rgommers]

This seems pretty happy so far, modulo the triggering issues in some pipelines.

Extremely helpful! numpy/numpy#26191

Glad it's useful:) For any incompatibilities we encounter here, I'd be happy to extend the compat table.

Most packages needed at least a few minor source-level changes. Most packages in that table also have ABI incompatibilities for wheels on PyPI - that's not going to matter for Spack, so it's very well possible that builds pass here even though they're marked as incompatible in numpy/numpy#26191.

[adamjstewart]

Any idea what's wrong with the remaining build errors (Linux, OneAPI)?

h5py

ImportError: numpy/_core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so: undefined symbol: pow8_h

petsc4py

  In file included from src/petsc4py/PETSc.c:1242:
  src/petsc4py/include/petsc4py/numpy.h:5:10: fatal error: 'numpy/arrayobject.h' file not found
      5 | #include <numpy/arrayobject.h>
        |          ^~~~~~~~~~~~~~~~~~~~~
  1 error generated.

scipy

ImportError: numpy/_core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so: undefined symbol: pow8_h

[rgommers]

The h5py and scipy failures look the same, and unrelated to those packages. The numpy build with oneAPI seems broken. I can't locate it easily though - how do I navigate to the build of numpy itself?

[adamjstewart]

I can't locate it easily though - how do I navigate to the build of numpy itself?

By default, Spack deletes the build directory (stage). If you use:

spack install --keep-stage py-numpy@2.0.0 %oneapi

it won't delete the build directory. You can then run:

spack cd py-numpy@2.0.0 %oneapi

to go to the build directory. Best source of documentation on this is probably https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html#debugging-package-builds.

[rgommers]

I meant in CI. There doesn't seem to be a good way to navigate from the failing jobs to the ones they depend on. It's the py-numpy build in CI that is doing something strange here I think. I'm away from my Linux dev machine this week, so can't try locally.

[adamjstewart]

I meant in CI.

Ah sorry. I actually don't know how. @scottwittenburg @kwryankrattiger? We're trying to go from a failing build to its successful parent, but the successful parent is minimized because nothing changed in the last commit.

Workaround: most recent commit that rebuilt numpy: https://gitlab.spack.io/spack/spack/-/jobs/11627167

[rgommers]

Workaround: most recent commit that rebuilt numpy: https://gitlab.spack.io/spack/spack/-/jobs/11627167

That log is full of errors, it shouldn't be passing. Right at the top of the log is this:

    from . import overrides
  File "/home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh/morepadding/linux-ubuntu22.04-x86_64_v3/oneapi-2024.0.0/py-numpy-2.0.0-tfvgeycazgvoco6wao4i5nk6cpmmylo2/lib/python3.11/site-packages/numpy/_core/overrides.py", line 8, in <module>
    from numpy._core._multiarray_umath import (
ImportError: /home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh/morepadding/linux-ubuntu22.04-x86_64_v3/oneapi-2024.0.0/py-numpy-2.0.0-tfvgeycazgvoco6wao4i5nk6cpmmylo2/lib/python3.11/site-packages/numpy/_core/_multiarray_umath.cpython-311-x86_64-linux-gnu.so: undefined symbol: pow8_h

And after that every single test is failing.

---

pow8_h is only used in the SVML sources shipped with NumPy (https://github.com/numpy/SVML/blob/32bf2a98420762a63ab418aaa0a7d6e17eb9627a/linux/avx512/svml_z0_pow_d_ha.s#L91). I've never seen an issue there - my first guess is a OneAPI bug in AVX512 support. OneAPI isn't tested in NumPy CI unfortunately, due to its ultra-clunky packaging.

[adamjstewart]

Okay. I'll mark it as incompatible for now and we can refine that version/compiler bounds if/when we learn more.

[kwryankrattiger]

I don't know if I understand the question. Just because something isn't rebuilt in that CI run, the binaries that it uses in downstream jobs will be whatever is in the build cache for that hash. Sometimes you have to look back in a PR to find the job that built that specific binary, or it was built previously on develop.

[tldahlgren]

Confirmed the version sha256. Perused the rest and it LGTM.

[adamjstewart]

../meson.build:138:12: ERROR: Program 'pythran' not found or not executable

I know we had this problem before, but I don't remember how we solved it.

[rgommers]

../meson.build:138:12: ERROR: Program 'pythran' not found or not executable

I know we had this problem before, but I don't remember how we solved it.

Hmm, this job doesn't have an outdated setuptools (like in #44967 (comment)), so maybe the problem is different.

There were a few PRs that touched py-pythran, but nothing jumps out. Somehow entrypoint handling is messed up again. Here is a recent log that builds py-pythran 0.16.1: https://gitlab.spack.io/spack/spack/-/jobs/11404844. It only tests imports; the entrypoint's existence isn't checked in the tests run by Pythran.

[adamjstewart]

It only tests imports; the entrypoint's existence isn't checked in the tests run by Pythran.

How would I test this on the command line? We can add a test to the package.py so that this is properly tested, although last I checked Python testing was broken (@tldahlgren)

Let's wait to see if my fix in #44967 magically helps for some unknown reason.

[rgommers]

Let's wait to see if my fix in #44967 magically helps for some unknown reason.

Looks like it did:)

How would I test this on the command line?

It's a matter of invoking pythran directly (e.g., pythran --version), rather than only importing it as a Python package.