CI: Test NumPy against OpenBLAS weekly builds

[honno]

This PR tests NumPy against OpenBLAS weekly builds, as well as stable builds like before.

Things to note:

• linux_meson.yml is updated to use the pre-existing cibw_before_build.sh/openblas_support.py scripts. These scripts are updated to support the weekly OpenBLAS builds, mirroring the changes from CI: Add prerelease build with 64bit OpenBLAS nightly scipy/scipy#18889 (testing SciPy against OpenBLAS weeklies). It feels weird to almost perfectly duplicate these scripts, and not use things like getopt, but it looks like they should be made redundant in the future anywho (CI: Add prerelease build with 64bit OpenBLAS nightly scipy/scipy#18889 (comment)).

• Before linux_meson.yml got OpenBLAS via apt-get install libopenblas-serial-dev. This PR currently pulls OpenBLAS from multibuild-wheels-staging... maybe there's value in testing the Ubuntu package too?

• We're hoping to get these stable and weekly builds that live in multibuild-wheels-staging to scientific-python-nightly-wheels Upload to scientific-python-nightly-wheels, change nightly frequency to weekly MacPython/openblas-libs#111, which looks like it could happen very soon. That PR got merged, so we can wait on it updating to make sure things work but should be good to go. See also CI: Change OpenBLAS nightly wheel location scipy/scipy#18942

cc @mattip @steppi @rgommers

[rgommers]

@honno the failures look relevant.

[mattip]

The macos-arm64 wheel build failure is due to the new code, it seems the getopts parsing is not right on macOS.

[steppi]

The macos-arm64 wheel build failure is due to the new code, it seems the getopts parsing is not right on macOS.

Right, this is why I reverted using getopt in the SciPy version of this PR scipy/scipy#18889 (comment)

[honno]

The macos-arm64 wheel build failure is due to the new code, it seems the getopts parsing is not right on macOS.

Right, this is why I reverted using getopt in the SciPy version of this PR scipy/scipy#18889 (comment)

Fixed, all green now 😅

[honno]

Per Matti's suggestions I've simplified a few things... and we should actually be testing against the weekly builds of OpenBLAS now 😅

[mattip]

Two last things:

• the jobs are called "meson_spin (true)" and "meson_spin (false)". Could you think of a way to get better names?
• Is there a check you can point to that the correct OpenBLAS version is being used? The azure builds use this, which might need to grow a --nightly option that would verify that the OpenBLAS version is not the one in the ``OPENBLAS_LONG` value:

  numpy/azure-pipelines.yml

  Lines 180 to 185 in 511f77d

  	- bash: |
  	python -m pip install threadpoolctl
  	python tools/openblas_support.py --check_version
  	displayName: 'Verify OpenBLAS version'
  	condition: eq(variables['USE_OPENBLAS'], '1')

[honno]

the jobs are called "meson_spin (true)" and "meson_spin (false)". Could you think of a way to get better names?

Yep, so TIL name attribute in jobs determines the... name, and we can do a hacky condition in a GitHub expression (see actions/runner#409 (comment)) to change it depending on say matrix values. Currently we get

Test Linux (stable OpenBLAS)
Test Linux (nightly OpenBLAS)

Is there a check you can point to that the correct OpenBLAS version is being used?

I played around with things, no success so far. Notably openblas_support.py::test_version() uses import numpy, which gives us an ImportError, seemingly by virtue of how NumPy is built in this workflow. Also its a pain to explore/debug how test_version() works locally. Might write a separate script for now that just prints the version.

[mattip]

Notably openblas_support.py::test_version() uses import numpy, which gives us an ImportError, seemingly by virtue of how NumPy is built in this workflow.

Since this is using the spin workflow, you will need to use spin python ... to execute the check

[honno]

which might need to grow a --nightly option that would verify that the OpenBLAS version is not the one in the OPENBLAS_LONG value:

The OpenBLAS nightly build seemingly has the same "version" as the stable build, looking at this run

ValueError: nightly OpenBLAS version should not be 0.3.23.dev, got [{'user_api': 'blas', 'internal_api': 'openblas', 'num_threads': 2, 'prefix': 'libopenblas', 'filepath': '/usr/lib/libopenblasp-r0.3.23.dev.so', 'version': '0.3.23.dev', 'threading_layer': 'pthreads', 'architecture': 'SkylakeX'}]

I checked against OPENBLAS_V, as thats what openblas_support.py::test_version() uses—I might of misunderstand you though. I assume the version attribute (however threadpoolctl gets it) would be whatever the latest released version is anywho. I'm not sure what to look for really...

Reproducing building with nightly OpenBLAS builds locally has been a huge time sink already, been hard to experiment 🙃

[rgommers]

That test_version check is horrible, I suggest we ignore it. Looking at the CI logs, it's clear that the "stable" job does python tools/openblas_support.py while the "nightly" job does python tools/openblas_support.py --nightly. If we'd just uncomment a print statement in download_openblas to verify the exact artifact we're downloading, we should be all good here.

[honno]

If we'd just uncomment a print statement in download_openblas to verify the exact artifact we're downloading, we should be all good here.

Introduced a new comment as wasn't sure what to do with # print(f"Downloading {length} from {filename}", file=sys.stderr). Looks like we're downloading the right files!

https://github.com/numpy/numpy/actions/runs/5716123665/job/15486923504?pr=24250#step:4:63
https://github.com/numpy/numpy/actions/runs/5716123665/job/15486923255?pr=24250#step:4:63

[rgommers]

The Circle CI build wasn't touched in this PR and is only failing because this branch is behind. The Cirrus CI failures are unrelated too. Still, a few too many failures to confidently merge - let's retry that now (I merged main into this PR).

[rgommers]

Okay, the intel_spr_sde_test failures are clearly unrelated, and the rest all looks good now. In it goes. Thanks @honno, @mattip & @steppi!