Update SciPy version #949

pasdeloup · 2023-08-31T09:50:42Z

A small step forward for #648 , I could update the existing patch and reach SciPy 1.6.3
Tested with Python 3.9 and arm64-v8a ABI

pasdeloup · 2023-08-31T10:05:12Z

Trying to go further, 1.6.3 version seems reachable!

pasdeloup · 2023-08-31T11:05:41Z

For versions 1.7.x and 1.8.x I still have issues with numpy

mhsmith · 2023-08-31T14:11:46Z

I put quotation marks around all the version numbers to prevent things like 0.10 being parsed as 0.1, so I've merged that into this PR. This also picks up all the changes mentioned in #648 (comment), so please let me know if that causes any trouble.

pasdeloup · 2023-09-01T07:34:53Z

Still working for me, I just had to reinstall requirements to get pypi_simple (and maybe others).

pasdeloup · 2023-09-01T10:42:27Z

Sorry, tests are not passing with 1.6.3 because of requirements:
Requires-Dist: numpy (<1.23.0,>=1.16.5)
but we build with numpy 1.23.3

… release: scipy/scipy#12862)

pasdeloup · 2023-09-01T12:59:25Z

Ok now tests OK

pasdeloup · 2023-09-04T11:03:26Z

Or maybe better to downgrade numpy to a <1.23.0,>=1.16.5 version, but which one ? We also need to provide it

mhsmith · 2023-09-04T13:58:14Z

Or maybe better to downgrade numpy to a <1.23.0,>=1.16.5 version

Yes, it's probably best to respect the limits declared by the package, even if they added them for questionable reasons.

Or maybe better to downgrade numpy to a <1.23.0,>=1.16.5 version, but which one ? We also need to provide it

1.19.5 is the obvious choice:

We already have it for Python 3.8, and it'll probably be the last version we release for 3.8 because of #570.
It didn't require any patch changes when we last updated the version, so building it for Python 3.9 shouldn't be a problem.
It's not compatible with Python 3.10 or later, but neither is SciPy 1.6.3.

…fore lib release: scipy/scipy#12862)" This reverts commit c27a650.

pasdeloup · 2023-09-05T10:06:56Z

Seemed obvious at first but can't make it build with 1.19.5, not sure why... :(

And as this version is not built yet for Python 3.9 (and still need some patch-fixing because of 52cef01 ) we are free to use another.

I propose 1.20.3 for Python 3.9
Seems a good choice according to https://docs.scipy.org/doc/scipy/dev/toolchain.html

And tests are passing

pasdeloup · 2023-09-05T11:42:33Z

By the way, to build numpy 1.20.3, just need to remove last update from the patch.
Could be great to allow keeping multiple patches, one per version.

mhsmith · 2023-09-05T12:14:05Z

Was this the error you received when building NumPy 1.19.5 for Python 3.9? I got this on x86 and armeabi-v7a, but not on x86_64 and arm64-v8a, which built successfully. So it's probably a 32 vs 64-bit issue.

And unfortunately it looks like 1.20.3 has the same problem.

building extension "numpy.core._multiarray_umath" sources
Traceback (most recent call last):
  File "/home/smith/.venv/pypi/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/home/smith/.venv/pypi/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/smith/.venv/pypi/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/env/lib/python3.9/site-packages/setuptools/build_meta.py", line 211, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/env/lib/python3.9/site-packages/setuptools/build_meta.py", line 197, in _build_with_temp_dir
    self.run_setup()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/env/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/env/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 512, in <module>
    setup_package()
  File "setup.py", line 504, in setup_package
    setup(**metadata)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/src/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/env/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 290, in run
    self.run_command('build')
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/src/numpy/distutils/command/build.py", line 40, in run
    old_build.run(self)
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/smith/.pyenv/versions/3.9.12/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/src/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/src/numpy/distutils/command/build_src.py", line 161, in build_sources
    self.build_extension_sources(ext)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/src/numpy/distutils/command/build_src.py", line 318, in build_extension_sources
    sources = self.generate_sources(sources, ext)
  File "/home/smith/git/chaquo/chaquopy/server/pypi/packages/numpy/build/1.19.5/cp39-cp39-android_21_x86/src/numpy/distutils/command/build_src.py", line 378, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 436, in generate_config_h
    moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
  File "numpy/core/setup.py", line 47, in check_types  
    out = check_types(*a, **kw)
  File "numpy/core/setup.py", line 292, in check_types 
    raise SystemError(
SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.

pasdeloup · 2023-09-06T08:41:11Z

Nope, numpy 1.19.5 is building, but Scipy is not when using this version as a dependency
I will check 32 bits ABIs

pasdeloup · 2023-09-06T09:19:25Z

Numpy 1.23.3 doesn't build anymore either on armeabi-v7a
The issue is due to 52cef01
We can build both 1.23.3 and 1.20.3 when reverting it

pasdeloup · 2023-09-07T09:09:51Z

Ok found, my mistake : we should not remove last patch, but alter it temporarily to build 1.20.3, the error is due to setuptools version

--- src-original/pyproject.toml	2022-09-09 13:36:30.619459000 +0000
+++ src/pyproject.toml	2023-08-31 11:38:08.304445745 +0000
@@ -1,7 +1,8 @@
 [build-system]
 # Minimum requirements for the build system to execute.
 requires = [
-    "setuptools<49.2.0",
+    # Chaquopy: was 59.2.0, which failed to detect modf and frexp for some reason.
+    "setuptools==63.0.0",
     "wheel<=0.35.1",
     "Cython>=0.29.21,<3.0",  # Note: keep in sync with tools/cythonize.py
 ]

So Numpy 1.23.3 is still building on armeabi-v7a with the existing patch.

mhsmith · 2023-09-09T15:37:26Z

Thanks, I'll try that now.

Could be great to allow keeping multiple patches, one per version.

Yes, but I'd prefer not to invest time into replicating yet another conda-build feature, because our long-term plan is to migrate to using conda-build directly.

mhsmith · 2023-09-09T22:59:50Z

I've built and released NumPy 1.20.3 for Python 3.9, and saved the recipe in #957. But now SciPy 1.6.3 is also failing on the 32-bit ABIs, with the error:

creating build/temp.linux_i686-3.9/scipy/special
creating build/temp.linux_i686-3.9/scipy/special/cephes
compile options: '-I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/src/scipy/special -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x
86/requirements/numpy/core/include -I/home/smith/.pyenv/versions/3.9.12/include/python3.9 -Iscipy/_lib -Iscipy/_build_utils/src -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/req
uirements/numpy/core/include -Ibuild/src.linux_i686-3.9/numpy/distutils/include -c'
extra options: '-msse -msse2'
i686-linux-android21-clang: scipy/special/cephes/airy.c
i686-linux-android21-clang: scipy/special/cephes/dawsn.c
In file included from scipy/special/cephes/airy.c:52:  
In file included from scipy/special/cephes/mconf.h:56: 
In file included from /home/smith/.pyenv/versions/3.9.12/include/python3.9/Python.h:50:
/home/smith/.pyenv/versions/3.9.12/include/python3.9/pyport.h:741:2: error: "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
...
error: Command "/home/smith/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/i686-linux-android21-clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/requirements/chaquopy/include -idirafter /home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/requirements/chaquopy/include/python3.9 -fPIC -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/src/scipy/special -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/requirements/numpy/core/include -I/home/smith/.pyenv/versions/3.9.12/include/python3.9 -Iscipy/_lib -Iscipy/_build_utils/src -I/home/smith/git/chaquo/chaquopy/server/pypi/packages/scipy/build/1.6.3/cp39-cp39-android_21_x86/requirements/numpy/core/include -Ibuild/src.linux_i686-3.9/numpy/distutils/include -c scipy/special/cephes/airy.c -o build/temp.linux_i686-3.9/scipy/special/cephes/airy.o -MMD -MF build/temp.linux_i686-3.9/scipy/special/cephes/airy.o.d -msse -msse2" failed with exit status 1

I think I know how to fix this, so I'll look into it tomorrow.

…wrapper script

mhsmith · 2023-09-10T16:57:10Z

I think both of the 32-bit failures were caused by the fact that the monkey patches in sitecustomize.py, which were supposed to filter out include and library paths from outside of the build directory, didn't work on the older versions of setuptools which were requested by the pyproject.toml files of these NumPy and SciPy versions.

#935, which involved a newer version of setuptools, was probably caused by the same thing.

So I've moved these directory filters into a compiler wrapper script instead. This means it'll take effect on all build systems, both Python and non-Python.

However, for Python packages, it would still be a good idea to look into using crossenv, so the invalid paths don't get put on the command line in the first place. This should also allow one of the remaining monkey patches in sitecustomize.py to be removed.

mhsmith · 2023-09-10T16:58:15Z

I've removed a couple of things from the patch which weren't needed anymore, and reordered the files so they're still in alphabetical order.

SciPy 1.6.3 is now in the public repository for Python 3.9. Thanks again!

Update Scipy to 1.5.4

9bb7508

pasdeloup changed the title ~~Update SciPy to 1.5.4~~ Update SciPy version Aug 31, 2023

Update to SciPy 1.6.3

27cc695

Merge branch 'master' into pr-scipy_1.5.4

fabf305

Patch to allow numpy>=1.23 (restriction was a safety added before lib…

c27a650

… release: scipy/scipy#12862)

pasdeloup added 2 commits September 5, 2023 11:00

Revert "Patch to allow numpy>=1.23 (restriction was a safety added be…

555ef80

…fore lib release: scipy/scipy#12862)" This reverts commit c27a650.

Use numpy 1.20.3 for Python 3.9

e205126

Merge branch 'master' into pr-scipy_1.5.4

48a3df2

mhsmith mentioned this pull request Sep 9, 2023

NumPy 1.20.3 #957

Closed

mhsmith added 2 commits September 10, 2023 16:46

Move include and lib path filtering from sitecustomize to a compiler …

fdc7629

…wrapper script

SciPy 1.6.3 tests passing on all ABIs with Python 3.9

cf3ed29

mhsmith merged commit d1f3567 into chaquo:master Sep 10, 2023

mhsmith mentioned this pull request Sep 10, 2023

Update scipy package version #648

Closed

pasdeloup mentioned this pull request Sep 12, 2023

Update SciPy package to 1.8.1 #960

Closed

mhsmith added the packages label Sep 13, 2023

mhsmith added the update label Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update SciPy version #949

Update SciPy version #949

pasdeloup commented Aug 31, 2023 •

edited

pasdeloup commented Aug 31, 2023

pasdeloup commented Aug 31, 2023

mhsmith commented Aug 31, 2023

pasdeloup commented Sep 1, 2023

pasdeloup commented Sep 1, 2023

pasdeloup commented Sep 1, 2023

pasdeloup commented Sep 4, 2023

mhsmith commented Sep 4, 2023

pasdeloup commented Sep 5, 2023

pasdeloup commented Sep 5, 2023

mhsmith commented Sep 5, 2023

pasdeloup commented Sep 6, 2023

pasdeloup commented Sep 6, 2023 •

edited

pasdeloup commented Sep 7, 2023

mhsmith commented Sep 9, 2023

mhsmith commented Sep 9, 2023 •

edited

mhsmith commented Sep 10, 2023

mhsmith commented Sep 10, 2023

Update SciPy version #949

Update SciPy version #949

Conversation

pasdeloup commented Aug 31, 2023 • edited

pasdeloup commented Aug 31, 2023

pasdeloup commented Aug 31, 2023

mhsmith commented Aug 31, 2023

pasdeloup commented Sep 1, 2023

pasdeloup commented Sep 1, 2023

pasdeloup commented Sep 1, 2023

pasdeloup commented Sep 4, 2023

mhsmith commented Sep 4, 2023

pasdeloup commented Sep 5, 2023

pasdeloup commented Sep 5, 2023

mhsmith commented Sep 5, 2023

pasdeloup commented Sep 6, 2023

pasdeloup commented Sep 6, 2023 • edited

pasdeloup commented Sep 7, 2023

mhsmith commented Sep 9, 2023

mhsmith commented Sep 9, 2023 • edited

mhsmith commented Sep 10, 2023

mhsmith commented Sep 10, 2023

pasdeloup commented Aug 31, 2023 •

edited

pasdeloup commented Sep 6, 2023 •

edited

mhsmith commented Sep 9, 2023 •

edited