Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade numpy to 1.26.0, setuptools to 68.2.2 #36123

Merged
merged 22 commits into from
Oct 8, 2023

Conversation

mkoeppe
Copy link
Member

@mkoeppe mkoeppe commented Aug 22, 2023

NumPy 1.26 provides Python 3.12 support and Cython 3 compatibility.

1.26.0 was released 2023-09-16.

As NumPy has changed its build system from setuptools-pinned-to-an-ancient-version to meson_python, we get rid of our ancient version of setuptools. The SPKGs setuptools and setuptools_wheel now ship the same (current) version.

Resolves #34816

📝 Checklist

  • The title is concise, informative, and self-explanatory.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@mkoeppe
Copy link
Member Author

mkoeppe commented Aug 23, 2023

New doctest failure - as seen in ttps://github.com/mkoeppe/sage/actions/runs/5943230862/job/16125795245#step:8:8985:

File "src/sage/cpython/getattr.pyx", line 410, in sage.cpython.getattr.dir_with_other_class
Failed example:
    cython("cdef class A:\n    cdef public int a")                            # optional - sage.misc.cython
Expected nothing
Got:
    WARNING: Overriding pythran description with argspec information for: numpy.random.binomial
    WARNING: Overriding pythran description with argspec information for: numpy.random.bytes
    WARNING: Overriding pythran description with argspec information for: numpy.random.chisquare
    WARNING: Overriding pythran description with argspec information for: numpy.random.choice
    WARNING: Overriding pythran description with argspec information for: numpy.random.dirichlet
    WARNING: Overriding pythran description with argspec information for: numpy.random.exponential
    WARNING: Overriding pythran description with argspec information for: numpy.random.f
    WARNING: Overriding pythran description with argspec information for: numpy.random.gamma
    WARNING: Overriding pythran description with argspec information for: numpy.random.geometric
    WARNING: Overriding pythran description with argspec information for: numpy.random.pareto
    WARNING: Overriding pythran description with argspec information for: numpy.random.gumbel
    WARNING: Overriding pythran description with argspec information for: numpy.random.poisson
    WARNING: Overriding pythran description with argspec information for: numpy.random.negative_binomial
    WARNING: Overriding pythran description with argspec information for: numpy.random.normal
    WARNING: Overriding pythran description with argspec information for: numpy.random.laplace
    WARNING: Overriding pythran description with argspec information for: numpy.random.logistic
    WARNING: Overriding pythran description with argspec information for: numpy.random.lognormal
    WARNING: Overriding pythran description with argspec information for: numpy.random.logseries
    WARNING: Overriding pythran description with argspec information for: numpy.random.power
    WARNING: Overriding pythran description with argspec information for: numpy.random.ranf
    WARNING: Overriding pythran description with argspec information for: numpy.random.randint
    WARNING: Overriding pythran description with argspec information for: numpy.random.random
    WARNING: Overriding pythran description with argspec information for: numpy.random.random_integers
    WARNING: Overriding pythran description with argspec information for: numpy.random.random_sample
    WARNING: Overriding pythran description with argspec information for: numpy.random.rayleigh
    WARNING: Overriding pythran description with argspec information for: numpy.random.sample
    WARNING: Overriding pythran description with argspec information for: numpy.random.standard_exponential
    WARNING: Overriding pythran description with argspec information for: numpy.random.standard_gamma
    WARNING: Overriding pythran description with argspec information for: numpy.random.standard_normal
    WARNING: Overriding pythran description with argspec information for: numpy.random.uniform
    WARNING: Overriding pythran description with argspec information for: numpy.random.weibull

@mkoeppe
Copy link
Member Author

mkoeppe commented Aug 23, 2023

New doctest failure (https://github.com/mkoeppe/sage/actions/runs/5943230862/job/16125795245#step:8:20690):

File "src/sage/tests/books/computational-mathematics-with-sagemath/graphique_doctest.py", line 135, in sage.tests.books.computational-mathematics-with-sagemath.graphique_doctest
Failed example:
    for k in srange(0, 10, 0.15):
          y = integrate.odeint(f, k, t)
          p += line(zip(t, flatten(y)))
Expected nothing
Got:
    doctest:warning
      File "/sage/src/bin/sage-runtests", line 162, in <module>
        err = DC.run()
      File "/sage/src/sage/doctest/control.py", line 1535, in run
        self.run_doctests()
      File "/sage/src/sage/doctest/control.py", line 1147, in run_doctests
        self.dispatcher.dispatch()
      File "/sage/src/sage/doctest/forker.py", line 2085, in dispatch
        self.parallel_dispatch()
      File "/sage/src/sage/doctest/forker.py", line 1980, in parallel_dispatch
        w.start()  # This might take some time
      File "/sage/src/sage/doctest/forker.py", line 2254, in start
        super().start()
      File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
        self._popen = self._Popen(self)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/context.py", line 281, in _Popen
        return Popen(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
        self._launch(process_obj)
      File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 71, in _launch
        code = process_obj._bootstrap(parent_sentinel=child_r)
      File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
        self.run()
      File "/sage/src/sage/doctest/forker.py", line 2226, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/sage/src/sage/doctest/forker.py", line 2556, in __call__
        doctests, extras = self._run(runner, options, results)
      File "/sage/src/sage/doctest/forker.py", line 2608, in _run
        result = runner.run(test)
      File "/sage/src/sage/doctest/forker.py", line 908, in run
        return self._run(test, compileflags, out)
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.tests.books.computational-mathematics-with-sagemath.graphique_doctest[55]>", line 3, in <module>
        p += line(zip(t, flatten(y)))
      File "/sage/src/sage/plot/line.py", line 311, in line
        return line2d(points, **kwds)
      File "/sage/src/sage/misc/decorators.py", line 657, in wrapper
        return func(*args, **kwds)
      File "/sage/src/sage/misc/decorators.py", line 496, in wrapper
        return func(*args, **options)
      File "/sage/src/sage/plot/line.py", line 619, in line2d
        xdata, ydata = xydata_from_point_list(points)
      File "/sage/src/sage/plot/plot.py", line 837, in xydata_from_point_list
        ydata.append(float(y))
      File "/usr/lib/python3.10/warnings.py", line 109, in _showwarnmsg
        sw(msg.message, msg.category, msg.filename, msg.lineno,
    :
    DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)

@mkoeppe mkoeppe marked this pull request as ready for review August 25, 2023 00:05
@mkoeppe mkoeppe mentioned this pull request Sep 3, 2023
5 tasks
@orlitzky
Copy link
Contributor

orlitzky commented Sep 20, 2023

install-requires.txt for setuptools needs a version bump too

@orlitzky
Copy link
Contributor

I needed this,

diff --git a/src/sage/repl/ipython_kernel/kernel.py b/src/sage/repl/ipython_kernel/kernel.py
index 73681716aa2..52635d27f1c 100644
--- a/src/sage/repl/ipython_kernel/kernel.py
+++ b/src/sage/repl/ipython_kernel/kernel.py
@@ -16,7 +16,13 @@ notebook or remote Jupyter sessions.
 # ***************************************************************************

 import sys
-from ipykernel.ipkernel import IPythonKernel
+import warnings
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore",
+                            message=r".*pkg_resources\.declare_namespace",
+                            category=DeprecationWarning)
+    from ipykernel.ipkernel import IPythonKernel
+
 from ipykernel.zmqshell import ZMQInteractiveShell
 from traitlets import Type

to avoid this,

File "src/sage/repl/ipython_kernel/kernel.py", line 48, in sage.repl.ipython_kernel.kernel.SageKernel.__init__
Failed example:
    from sage.repl.ipython_kernel.kernel import SageKernel
Expected nothing
Got:
    ...
    DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('pydevd_plugins.extensions.types')`.
    Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
**********************************************************************
1 item had failures:
   1 of   3 in sage.repl.ipython_kernel.kernel.SageKernel.__init__
    [12 tests, 1 failure, 0.87 s]

but otherwise it's working as advertised. Note that I was using setuptools-68.0.0, before you updated install-requires.txt.

@mkoeppe
Copy link
Member Author

mkoeppe commented Sep 21, 2023

Could you try if updating debugpy would remove this warning?

@orlitzky
Copy link
Contributor

Could you try if updating debugpy would remove this warning?

I don't think it's the version that's the problem, rather that Gentoo unbundles pydevd from debugpy. The warning ultimately comes from upstream pydevd (latest version): https://github.com/fabioz/PyDev.Debugger/blob/main/pydevd_plugins/extensions/types/__init__.py

@mkoeppe
Copy link
Member Author

mkoeppe commented Sep 21, 2023

debugpy apparently has a fix for this -microsoft/debugpy@00bc513

but it's fine if you want to push this filterwarnings to the branch here.

@orlitzky
Copy link
Contributor

debugpy apparently has a fix for this -microsoft/debugpy@00bc513

but it's fine if you want to push this filterwarnings to the branch here.

Gentoo only has debugpy-1.6.6 so I'd like to support it if it's not too much trouble. I got permission denied trying to push to your branch, but here's a patch you can git am:

From 75b57552494e4578f3edcdbc4b55b9ad74a4c58b Mon Sep 17 00:00:00 2001
From: Michael Orlitzky <michael@orlitzky.com>
Date: Thu, 21 Sep 2023 17:20:32 -0400
Subject: [PATCH 1/1] src/sage/repl/ipython_kernel/kernel.py: hide a PEP 420
 warning

Hide a warning that stems from using upstream (unbundled) pydevd with
debugpy. New versions of debugpy work around it, but it's better if we
can support older versions for a while.
---
 src/sage/repl/ipython_kernel/kernel.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/src/sage/repl/ipython_kernel/kernel.py b/src/sage/repl/ipython_kernel/kernel.py
index 73681716aa2..db7554eed59 100644
--- a/src/sage/repl/ipython_kernel/kernel.py
+++ b/src/sage/repl/ipython_kernel/kernel.py
@@ -16,7 +16,18 @@ notebook or remote Jupyter sessions.
 # ***************************************************************************
 
 import sys
-from ipykernel.ipkernel import IPythonKernel
+import warnings
+with warnings.catch_warnings():
+    # When upstream pydevd (as opposed to the bundled version) is used
+    # with debugpy, a PEP 420 warning is emitted. Debugpy and/or
+    # pydevd will eventually work around this, but as of September
+    # 2023, hiding the warning gives us more flexibility in the
+    # versions of those packages that we can accept.
+    warnings.filterwarnings("ignore",
+                            message=r".*pkg_resources\.declare_namespace",
+                            category=DeprecationWarning)
+    from ipykernel.ipkernel import IPythonKernel
+
 from ipykernel.zmqshell import ZMQInteractiveShell
 from traitlets import Type
 
-- 
2.41.0

Hide a warning that stems from using upstream (unbundled) pydevd with
debugpy. New versions of debugpy work around it, but it's better if we
can support older versions for a while.
@mkoeppe mkoeppe changed the title Upgrade numpy to 1.26.0 Upgrade numpy to 1.26.0, setuptools to 68.2.2 Sep 22, 2023
@mkoeppe
Copy link
Member Author

mkoeppe commented Sep 27, 2023

All dependencies are merged; needs final review

@github-actions
Copy link

Documentation preview for this PR (built with commit 2eb2250; changes) is ready! 🎉

@orlitzky
Copy link
Contributor

I was hoping the usual suspects would drop in and test this on a few more distributions, but it's been here for over a month, so.

@mkoeppe
Copy link
Member Author

mkoeppe commented Sep 29, 2023

Thanks!

vbraun pushed a commit to vbraun/sage that referenced this pull request Oct 1, 2023
    
<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

NumPy 1.26 provides Python 3.12 support and Cython 3 compatibility.

1.26.0 was released 2023-09-16.

As NumPy has changed its build system from setuptools-pinned-to-an-
ancient-version to meson_python, we get rid of our ancient version of
`setuptools`. The SPKGs `setuptools` and `setuptools_wheel` now ship the
same (current) version.

- [x] Check portability run:
https://github.com/mkoeppe/sage/actions/runs/5959239800
- [ ] Check SAGE_FAT_BINARY

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
Resolves sagemath#34816
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies
- Depends on sagemath#36112 (for the `meson_python` upgrades, merged here)
- Depends on sagemath#35404 (merged here as
part of above)
- Depends on sagemath#35810 (merged here as part of above)
- Depends on sagemath#36110 (merged here)
- Depends on sagemath#36263 (merged here)
- Depends on sagemath#36238 (merged here)
- Depends on sagemath#36255 (merged here)
<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36123
Reported by: Matthias Köppe
Reviewer(s):
@vbraun
Copy link
Member

vbraun commented Oct 8, 2023

This breaks Debian 10 "Buster" which is still on the supported platform list:

  ../../meson.build:30:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4

@vbraun
Copy link
Member

vbraun commented Oct 8, 2023

OK I see that it was removed by one of the dependencies

@vbraun vbraun merged commit 868fefa into sagemath:develop Oct 8, 2023
18 of 52 checks passed
@mkoeppe mkoeppe deleted the numpy_1_26 branch May 4, 2024 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update numpy to 1.26
3 participants