Difficulties when mandating NumPy availability

[knoepfel]

Phlex v0.1.0 includes an optional dependency on NumPy. If the intention is to use NumPy to translate between (e.g.) C++ std::vector<T> types and Python types, then NumPy should be a required dependency.

Unfortunately, our CI system does not currently build (or test) with NumPy. This is evidenced by the lack of py:vec test in the PR that incorporated Python support.

An attempt to mandate NumPy using CMake is shown in PR #219. The failure to import NumPy is explicit when running the tests. Specifically:

 52/61 Test #54: py:vec ..................................***Failed    0.16 sec
  Using configuration file: /__w/phlex/phlex/phlex-src/test/python/pyvec.jsonnet
  [2026-01-07 19:03:44.324] [info] Number of worker threads: 4
  ModuleNotFoundError: No module named 'numpy'
  ImportError: numpy._core.multiarray failed to import
  [2026-01-07 19:03:44.409] [debug] Routed 0 messages in 0 microseconds (-nan microseconds per message)
  [2026-01-07 19:03:44.409] [info] 
  [2026-01-07 19:03:44.409] [info] CPU time: 0.06370s  Real time: 0.08574s  CPU efficiency:  74.30%
  [2026-01-07 19:03:44.409] [info] Max. RSS: 30.628 MB
  build with numpy support, but numpy not importable

We should mandate the use of NumPy and solve this problem.

[knoepfel]

I've learned more. If I activate the CI container's environment, I see:

root@2e866cf18b4f:~/build-phlex# spack find py-numpy
==> In environment /opt/spack-environments/phlex-ci
==> 9 root specs
-- no arch / no compilers ---------------------------------------
[+] catch2  [+] cmake  [+] lcov  [+] llvm@21.1.4:+link_llvm_dylib+llvm_dylib+zstd targets=x86  [+] ninja   -  phlex  [+] py-cmake-format  [+] py-gcovr  [+] py-pip

-- linux-debian13-x86_64_v3 / %c,cxx=gcc@15.2.0 -----------------
py-numpy@2.3.5
==> 1 installed package
==> 0 concretized packages to be installed (show with `spack find -c`)

So py-numpy is installed in the CI container...it's just not available:

root@2e866cf18b4f:~/build-phlex# spack python -c 'import numpy'
Traceback (most recent call last):
  File "<input>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'

If I then explicitly load it, I see the following:

root@2e866cf18b4f:~/build-phlex# spack load py-numpy
root@2e866cf18b4f:~/build-phlex# spack python -c 'import numpy'
Traceback (most recent call last):
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/_core/__init__.py", line 22, in <module>
    from . import multiarray
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/_core/multiarray.py", line 11, in <module>
    from . import _multiarray_umath, overrides
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/_core/overrides.py", line 5, in <module>
    from numpy._core._multiarray_umath import (
    ...<3 lines>...
    )
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/__init__.py", line 112, in <module>
    from numpy.__config__ import show_config
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
    ...<3 lines>...
    )
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/_core/__init__.py", line 48, in <module>
    raise ImportError(msg) from exc
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.13 from "/usr/bin/python3"
  * The NumPy version is: "2.3.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy._core._multiarray_umath'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.5-qczi2aqli7yp2w27zwczzidj5rhvbe43/lib/python3.14/site-packages/numpy/__init__.py", line 117, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

This should be sorted out.