Installation issue: Cython compiler 'cython' cannot compile programs when building py-scipy

[giordano]

Steps to reproduce the issue

I think I'm running into two different issues with compiling py-scipy. First of all, due to serge-sans-paille/pythran#2101 I'm running into the error

AttributeError: type object 'PatternTransform' has no attribute 'typePatterns'. Did you mean: 'CallPatterns'?

reported at scipy/scipy#18390. I believe this one should be worked around by adding

    # https://github.com/serge-sans-paille/pythran/issues/2101
    conflicts("python@3.11:", when="@:0.12.1")

to the py-pythran recipe in Spack. However, after that I still get another compilation error

$ spack -e . spec py-scipy                                                                                                                                     
Input spec                                                                                                                                                                                                        
--------------------------------                                                                                                                                                                                  
 -   py-scipy                                                                                                                                                                                                     
                                                                                                                                                                                                                  
Concretized                                                                                                                                                                                                       
--------------------------------                                                                                                                                                                                  
 -   py-scipy@1.11.4%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512                                                                                                                           
[+]      ^openblas@0.3.24%gcc@12.3.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-rhel7-skylake_avx512                              
[+]          ^gmake@4.4.1%gcc@12.3.0~guile build_system=generic arch=linux-rhel7-skylake_avx512                                                                                                                   
[+]          ^perl@5.38.0%gcc@12.3.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rhel7-skylake_avx512                                                                        
[+]              ^berkeley-db@18.1.40%gcc@12.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel7-skylake_avx512                                                                      
[+]      ^pkgconf@1.9.5%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512                                                                                                                         
[+]      ^py-cython@0.29.36%gcc@12.3.0 build_system=python_pip patches=c4369ad arch=linux-rhel7-skylake_avx512
[+]          ^py-setuptools@63.4.3%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^py-meson-python@0.13.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^meson@1.2.1%gcc@12.3.0 build_system=python_pip patches=0f0b1bd,ae59765 arch=linux-rhel7-skylake_avx512
[+]          ^py-pyproject-metadata@0.7.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]              ^py-packaging@23.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^py-tomli@2.0.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]              ^py-flit-core@3.9.0%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]      ^py-numpy@1.24.3%gcc@12.3.0 build_system=python_pip patches=873745d arch=linux-rhel7-skylake_avx512
[+]      ^py-pip@23.1.2%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^py-pybind11@2.11.0%gcc@12.3.0~ipo build_system=cmake build_type=Release generator=ninja arch=linux-rhel7-skylake_avx512
[+]          ^cmake@3.27.7%gcc@12.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel7-skylake_avx512
[+]              ^curl@8.4.0%gcc@12.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel7-skylake_avx512
[+]                  ^nghttp2@1.57.0%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^ninja@1.11.1%gcc@12.3.0+re2c build_system=generic arch=linux-rhel7-skylake_avx512
[+]              ^python@3.11.6%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rhel7-skylake_avx512
[+]              ^re2c@2.2%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^py-pythran@0.12.0%gcc@12.3.0 build_system=python_pip patches=5ccf989 arch=linux-rhel7-skylake_avx512
[+]          ^py-beniget@0.4.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^py-gast@0.5.4%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^py-ply@3.11%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]      ^py-wheel@0.41.2%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^python@3.10.13%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060 arch=linux-rhel7-skylake_avx512
[+]          ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-rhel7-skylake_avx512
[+]              ^diffutils@3.9%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^expat@2.5.0%gcc@12.3.0+libbsd build_system=autotools arch=linux-rhel7-skylake_avx512
[+]              ^libbsd@0.11.7%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]                  ^libmd@1.0.4%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^gdbm@1.23%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^gettext@0.22.3%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel7-skylake_avx512
[+]              ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-rhel7-skylake_avx512
[+]              ^libxml2@2.10.3%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-rhel7-skylake_avx512
[+]              ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-rhel7-skylake_avx512
[+]                  ^pigz@2.7%gcc@12.3.0 build_system=makefile arch=linux-rhel7-skylake_avx512
[+]                  ^zstd@1.5.5%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel7-skylake_avx512
[+]          ^libffi@3.4.4%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^libxcrypt@4.4.35%gcc@12.3.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-rhel7-skylake_avx512
[+]          ^ncurses@6.4%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^openssl@3.1.3%gcc@12.3.0~docs+shared build_system=generic certs=mozilla arch=linux-rhel7-skylake_avx512
[+]              ^ca-certificates-mozilla@2023-05-30%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]          ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-rhel7-skylake_avx512
[+]          ^sqlite@3.43.2%gcc@12.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^util-linux-uuid@2.38.1%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^xz@5.4.1%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-rhel7-skylake_avx512
[+]          ^zlib-ng@2.1.4%gcc@12.3.0+compat+opt build_system=autotools arch=linux-rhel7-skylake_avx512

Error message

Error message

  ../meson.build:1:0: ERROR: Cython compiler 'cython' cannot compile programs

  A full log can be found at /lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/build_stage/spack-stage-py-scipy-1.11.4-eys3x6zg35sw4g2wijiyofnknd2nyjx6/spack-src/build/meson-logs/meson-log.tx
t
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Exception information:
Traceback (most recent call last):
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/operations/build/metadata.p
y", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 713, i
n prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", li
ne 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", li
ne 311, in _call_hook
    self._subprocess_runner(
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/utils/subprocess.py", line 
252, in runner
    call_subprocess(
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/utils/subprocess.py", line 
224, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: Preparing metadata (pyproject.toml) exited with 1

Information on your system

• Spack: 0.21.1 (e30feda)
• Python: 3.6.8
• Platform: linux-rhel7-skylake_avx512
• Concretizer: clingo

Additional information

spack-build-env.txt
spack-build-out.txt

CC: @adamjstewart @rgommers

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
• I have uploaded the build log and environment files
• I have searched the issues of this repo and believe this is not a duplicate

[rgommers]

Adding the conflict seems correct to me.

to the py-pythran recipe in Spack. However, after that I still get another compilation error

The concretization above and the attached spack-build-env.txt still show Pythran 0.12.0 being used for Python 3.10. It's not clear from the logs what the actual problem is - could be the same one, or a different one.

Pythran 0.12.0 won't be tested well (if at all) with SciPy 1.11.4, since the metadata in https://github.com/scipy/scipy/blob/maintenance/1.11.x/pyproject.toml shows:

    # Reason for `<`: future-proofing (0.14.0 released and known to work)
    "pythran>=0.12.0,<0.15.0",

pip is always going to be choosing the highest version in that range, so I would not be surprised if 0.12.0 turns out to have an issue. Could you try bumping the lower bound for py-pythran in the py-scipy issue until your problem goes away?

[giordano]

The concretization above and the attached spack-build-env.txt still show Pythran 0.12.0 being used for Python 3.10.

Yeah, this is part of a larger environment (the goal is to try and build py-alphafold@2.2.4 +cuda, but it seems to be an almost impossible task), so there are other constraints involved. If I try to do py-alphafold@2.2.4 +cuda ^py-pythran@0.12.2 the environment refuses to concretize before of other conflicts somewhere else.

Could you try bumping the lower bound for py-pythran in the py-scipy issue until your problem goes away?

Sorry, I'm not sure what you're suggesting exactly. Are you suggesting to change

spack/var/spack/repos/builtin/packages/py-scipy/package.py

Line 81 in 763f444

depends_on("py-pythran@0.12:", when="@1.10:", type="build")

to require, for example, py-pythran@0.12.2:? I'm not sure that'd work because of the problem above with other py-alphafold deps.

[adamjstewart]

You can also try #42213 which adds newer versions of many of these packages.

[rgommers]

to require, for example, py-pythran@0.12.2:? I'm not sure that'd work because of the problem above with other py-alphafold deps.

Yes, that is what I was suggesting indeed. py-pythran is only a build dependency of py-scipy, so I was hoping that the separate per-package concretization of build dependencies would avoid any conflicts here. Not sure if that feature is opt-in or if it works differently than I imagined if there are still conflicts.

[adamjstewart]

Not sure if that feature is opt-in or if it works differently than I imagined if there are still conflicts.

See https://spack.readthedocs.io/en/latest/build_settings.html#duplicate-nodes

We would need to add tags = ["build-tools"] to the package to make it the default behavior, and only in Spack 0.21+

[giordano]

Ok, the incompatibility of py-pythran@0.12.2 with py-alphafold@2.2.4 may have been somehow (newer versions of other packages?) fixed in the develop branch of Spack, because in spack v0.21.1 I can't concretize spack spec --fresh py-alphafold@2.2.4 ^py-scipy@1.11.4 ^py-pythran@0.12.2, while this spec is successfully concretized in develop. However I can't really test building py-alphafold@2.2.4 is successful on develop, my current focus is trying to get a stack working with a stable version of spack and backporting dozens of recipes of python packages isn't feasible (I'm backporting only the few ones I can handle).

I'll open a PR to add the incompatibility of py-pythran@:0.12.1 with python@3.11: and will consider this issue fixed for the time being.

[giordano]

However I can't really test building py-alphafold@2.2.4 is successful on develop, my current focus is trying to get a stack working with a stable version of spack and backporting dozens of recipes of python packages isn't feasible (I'm backporting only the few ones I can handle).

Update: I moved to another machine (I'm not going to touch the production one at this point 🥲) and got py-alphafold@2.2.4%gcc@12.3.0+cuda cuda_arch=80 ^openmm@7.5.1+cuda cuda_arch=80 ^py-pythran@0.12.2 ^py-scipy@1.11.4 building successfully with spack@develop (1a1bbb8). I guess I will have to wait June to get a stable version of Spack for deploying py-alphafold on our systems 🙂