Tracking issue for further build system improvements

[rgommers]

After switching to Meson by default in gh-16187 and closing the RFC for that switch in gh-13615, here is a summary of next steps to improve our build system support. Roughly in order of priority:

• BLD: pip install build issue with meson in Ubuntu virtualenv #16300
• Improved BLAS/LAPACK detection/support (it works now, but only using pkg-config), see Improve BLAS/LAPACK support: dependency detection, UX, and ILP64 #17244 and Feature: MKL dependency mesonbuild/meson#2835
• Meson complains about an absolute include path #16312
• Support editable installs (in progress): Add support for editable wheels mesonbuild/meson-python#47
• ILP64 support. This is now still tested in one numpy.distutils based CI job, and not supported yet in the Meson-based build. See Improve BLAS/LAPACK support: dependency detection, UX, and ILP64 #17244
• Passing through SciPy-specific build options when using pip, pypa/build or similar: Passing through project-specific config options mesonbuild/meson-python#54
• Improve Cython support in Meson (not so urgent for SciPy, but important when other projects like scikit-learn are going to move to Meson):
  • Depfile support: Cython dependency handling in .pxd and .pxi files mesonbuild/meson#9049
  • After that, fix Cython does not respect dependency for generating .pyx.c file mesonbuild/meson#8961 (will allow a much simpler use of Cython in meson.build files)
  • In Cython, we need to not depend on __init__.py's anymore, which can be done by merging this PR: MRG: add --module-name argument to cython command cython/cython#4548
• Documented and tested (in CI) support for cross-compilation. It may work now, but hasn't been tested yet. See Tracking issue for cross-compilation needs and issues #14812.
• Get rid of the annoying warnings that we now see in the configure stage of the build: Cannot avoid 'more than one output' warning from generator depends mesonbuild/meson#9837. Done in BLD: fix Meson build warnings about multiple targets #17882.
• Ensure our runtime lower bound on numpy is >= that of the numpy we built against: UX for constraining build-time and runtime dependencies mesonbuild/meson-python#29. Note that the numpy.distutils based build never did this, but it's a correctness issue that can pop up in practice. EDIT: fixed by design in numpy now that we're building against 2.0.0.dev0. Once 1.25.0 is our lowest supported version, this is completely fixed.
• Build warnings in wheel build phase about "purelib or platlib" #16310
• Making dependency('numpy') work in Meson: Adding numpy as a dependency with custom lookup  mesonbuild/meson#9598. This will for us only result in a minor cleanup in meson.build files, but it's nice to have for other projects that are adopting Meson - one less thing to figure out for them. EDIT: this is important for cross-compilation!
• Don't depend on a shebang line in the Meson build (BUG: Build failure due to problems with shebang line in cythoner.py #17020)

[tylerjereddy]

(maybe build system, but not meson-specific, not sure) For non-editable python -m pip install . followed by make html-scipyorg I see the error below the fold working in a Python 3.10 virtualenv on Ubuntu. I'll try the old python setup.py install to see if that plays nicer with make html-scipyorg (this is just for local iteration/checking for relnotes errors).

mkdir -p build/html build/doctrees
LANG=C python3 -msphinx -WT --keep-going -t scipyorg  -b html -d build/doctrees  source build/html-scipyorg 
Running Sphinx v4.5.0

Traceback (most recent call last):
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/config.py", line 332, in eval_config_file
    exec(code, namespace)
  File "/home/treddy/github_projects/scipy/doc/source/conf.py", line 22, in <module>
    import scipy._lib.uarray as ua
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/__init__.py", line 148, in <module>
    from scipy._lib._ccallback import LowLevelCallable
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/_lib/_ccallback.py", line 1, in <module>
    from . import _ccallback_c
ImportError: cannot import name '_ccallback_c' from 'scipy._lib' (/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/_lib/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/cmd/build.py", line 272, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/application.py", line 202, in __init__
    self.config = Config.read(self.confdir, confoverrides or {}, self.tags)
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/config.py", line 165, in read
    namespace = eval_config_file(filename, tags)
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/config.py", line 345, in eval_config_file
    raise ConfigError(msg % traceback.format_exc()) from exc
sphinx.errors.ConfigError: There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/config.py", line 332, in eval_config_file
    exec(code, namespace)
  File "/home/treddy/github_projects/scipy/doc/source/conf.py", line 22, in <module>
    import scipy._lib.uarray as ua
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/__init__.py", line 148, in <module>
    from scipy._lib._ccallback import LowLevelCallable
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/_lib/_ccallback.py", line 1, in <module>
    from . import _ccallback_c
ImportError: cannot import name '_ccallback_c' from 'scipy._lib' (/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/_lib/__init__.py)


Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/sphinx/config.py", line 332, in eval_config_file
    exec(code, namespace)
  File "/home/treddy/github_projects/scipy/doc/source/conf.py", line 22, in <module>
    import scipy._lib.uarray as ua
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/__init__.py", line 148, in <module>
    from scipy._lib._ccallback import LowLevelCallable
  File "/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/_lib/_ccallback.py", line 1, in <module>
    from . import _ccallback_c
ImportError: cannot import name '_ccallback_c' from 'scipy._lib' (/home/treddy/python_virtual_envs/python_310_scipy_general/lib/python3.10/site-packages/scipy/_lib/__init__.py)

[rgommers]

Can you open as a new issue instead @tylerjereddy? I think I know what it is, and it needs a fix in meson-python; shouldn't be hard, but probably a release blocker for RC1 (EDIT: not sure actually, let's see).

[tylerjereddy]

Okie, I opened gh-16300 for that. In the meanwhile, I can iterate on release notes just fine with the old setuptools install.

[rgommers]

FYI, support for editable installs (i.e., pip install -e .) was just merged into meson-python: mesonbuild/meson-python#87. It's not released yet, but now would be a good time to test if this suits your needs if you like editable installs (IIRC @mdhaber, @tupui, @stefanv you all asked about this before). Docs at https://meson-python.readthedocs.io/en/latest/#editable-installs.

[tupui]

Youhou 🎉 testing this first thing tomorrow, thanks

[mdhaber]

Thanks. Does this mean that we can follow the instructions of the quickstart up to python dev.py, and instead of python dev.py we do python -m pip install -e .? Or do we need to manually do some of the other things that dev.py does (until dev.py has an editable build option)?

It says "Windows and MacOS are not yet fully supported, but that is actively being worked on." Do you think it's worth trying on Windows yet?

[rgommers]

Does this mean that we can follow the instructions of the quickstart up to python dev.py, and instead of python dev.py we do python -m pip install -e .?

Almost - quickstart until that point, and then use pip install -e . --no-build-isolation.

It says "Windows and MacOS are not yet fully supported, but that is actively being worked on." Do you think it's worth trying on Windows yet?

That should all work, and was not specific to the editable builds. I'll go delete that sentence.

[mdhaber]

Great. Suppose I'm starting with a fresh Windows machine - would I still need to set up compilers (from MSYS2 and Microsoft Visual Studio) as I did in the past, or will conda install ones that will work? And I've always needed to build OpenBLAS and set that up in site.cfg - anything like that, or should conda install a BLAS that will work?

[rgommers]

pip knows nothing about conda or compilers or non-PyPI dependencies, so that's all still unchanged. Editable installs are more limited than dev.py and certainly won't do anything extra. So the main reason to use them is if your IDE does not understand dev.py and assumes an editable install for Python projects. Note that the --no-build-isolation there is important, and your IDE almost certainly won't add that flag by default.

[tupui]

@rgommers I tried to play with this. Importing SciPy seems fine, but running tests raises some error

❯ pytest scipy/stats/tests/test_sensitivity_analysis.py
ImportError while loading conftest '/Users/tupui/Documents/Code/scipy/scipy/conftest.py'.
_pytest.pathlib.ImportPathMismatchError: ('scipy.conftest', '/Users/tupui/Documents/Code/scipy/.mesonpy/editable/install/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev/lib/python3.10/site-packages/scipy/conftest.py', PosixPath('/Users/tupui/Documents/Code/scipy/scipy/conftest.py'))

And trying to bypass the error with PY_IGNORE_IMPORTMISMATCH=1 leads to

ERROR  - ValueError: Plugin already registered: /Users/tupui/Documents/Code/scipy/.mesonpy/editable/install/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev/lib/python3.10/site-packages/scipy/conftest.py=<module 'scipy.conft...

I am trying to build all from a clean install. Maybe I had some leftovers (I did remove various pyc and caches)
EDIT: no still the same issues.

[tylerjereddy]

@tupui does it help if, after doing the editable install, you first move to another directory (i.e., cd /tmp) and then run tests there with something like python -m pytest --pyargs scipy?

[tupui]

@tylerjereddy this is working, but as the log shows it's using files in the /.mesonpy/editable folder and not the actual files.

❯ python -m pytest --pyargs scipy.stats.tests.test_qmc
========================================================================================================= test session starts =========================================================================================================
platform darwin -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/tupui
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, timeout-2.1.0, cov-3.0.0
collected 135 items                                                                                                                                                                                                                   

Documents/Code/scipy/.mesonpy/editable/install/opt/homebrew/Caskroom/mambaforge/base/envs/scipy-dev/lib/python3.10/site-packages/scipy/stats/tests/test_qmc.py .......................................s.s.s.s.s.s...sss........ [ 47%]
............................s.s.s.s.s...sss............................                                                                                                                                                         [100%]

=================================================================================================== 118 passed, 17 skipped in 1.90s ===================================================================================================

This is not good as this would prevent the debugger from working. I mean it's not improving thing, I can still debug if I put a breakpoint in the /.mesonpy/editable sources. Not really "in-place".

[rgommers]

This is not good as this would prevent the debugger from working. I mean it's not improving thing, I can still debug if I put a breakpoint in the /.mesonpy/editable sources. Not really "in-place".

Can we move this discussion to mesonbuild/meson-python#47? That's the right place, and an identical question got raised there.

[rgommers]

There's a new implementation for editable installs in meson-python ready for testing in mesonbuild/meson-python#279. If you're an editable install user, feedback directly on that PR would be quite useful.

[mdhaber]

Almost - quickstart until that point, and then use pip install -e . --no-build-isolation.

This meant to follow the pip quickstart until that point, I suppose?

pip knows nothing about conda or compilers or non-PyPI dependencies, so that's all still unchanged. Editable installs are more limited than dev.py and certainly won't do anything extra.

Will it be possible to extend dev.py to do editable installs with these extra features (i.e. it sounds like it is conda-aware)?

[rgommers]

Will it be possible to extend dev.py to do editable installs with these extra features (i.e. it sounds like it is conda-aware)?

Perhaps, not sure yet until we have some experience with how editable installs work under the hood.

[tylerjereddy]

@rgommers sorry if this is not quite the correct issue, but it is related to build system stuff--do you have any specific guideline on how I should treat the Python 3.12/bye-bye distutils situation with respect to planning the timeline for SciPy 1.11.0? Is there a summary somewhere of the things we likely need to properly handle 3.12 support (or, at least, the bare minimum)? Are we comfortable mostly proceeding "as usual" with the release timeline, and dealing with 3.12 when it is closer to being final in October?

From: https://peps.python.org/pep-0693/

3.12.0 beta 1: Monday, 2023-05-08 (No new features beyond this point.)

I suppose that's before our 1.11.0 release at least.

[andyfaff]

3.12 is in alpha, should we add it to CI now, or once it hits beta?

EDIT: FWIW I ran the test suite in a Linux VM with python3.12, all tests pass.

[rgommers]

Good question. I think we're okay to drop setup.py builds now. The only thing it still offers (beyond confusion) is the ILP64 build, which has a very small number of users (I'm not even sure if it's >0). And those can still use 1.10.x or the current state of main. I actually do need it to finish off support in Meson, which I'll need to do soon because NumPy does need ILP64 OpenBLAS.

For Python 3.12, I think it's too early. There are too many issues with Cython or build system stuff breaking, and there are no real upsides to start testing now (I don't think it'll catch any SciPy regressions). I'd start testing in June/July, not before.

until we have some experience with how editable installs work under the hood.

This works pretty smoothly now with meson-python's main branch right now by the way, and we should have it in a 0.13.0 release soon. So the IDE users can get their visual breakpoints and other goodies back.

Are we comfortable mostly proceeding "as usual" with the release timeline, and dealing with 3.12 when it is closer to being final in October?

I think it's healthy to have 1.11.x in good shape, so any potential backports will be small. Hence dropping setup.py now and updating our CI configs. Let me know if that sounds good, and if so I'll open a new issue with a migration plan? I think that'll involve wiping out Azure DevOps usage completely - 6/8 of our open issues with the CI label are due to Azure, and it's generally a pain to work with.

[h-vetinari]

FWIW, conda-forge on windows is not ready for meson builds. But I guess we'd survive if we dropped windows builds for the next release (though realistically, I'd probably try to patch it back in).

[andyfaff]

I looked at moving some of the jobs from Azure. My glances at the config file weren't easy on the eye - next to no comments in the files

[rgommers]

Ugh yes, forgot about the conda-forge Windows build - adding it to the list above. We'd need to hurry that one along. Okay, better plan: let's do the CI migration we need and move everything to Meson - but keep the setup.py files and two Windows CI jobs with setup.py, for the minimum and maximum Python versions.

I looked at moving some of the jobs from Azure. My glances at the config file weren't easy on the eye - next to no comments in the files

I'd rewrite them from scratch to be honest, based on only the info of what we're trying to test. We migrated Travis CI to Azure last time around and that made the migration quick but the end result pretty much unreadable.

[andyfaff]

If there's info on what we're trying to test I can get started on the transfer.

[rgommers]

Let me post on the most relevant issue for this topic, which is gh-15814.