Notebook 7, ipykernel 6.27, ipython 8.17

[mkoeppe]

This is now based on JupyterLab. There are many changes in dependencies.

We are trying to use platform-independent wheels when available, to avoid any Node.JS activity during our build.
Hence we remove our nodeenv, nodejs, jupyter_packaging, hatch_nodejs_version packages.

There is trouble on the horizon regarding our model of building everything from source: The latest versions of jsonschema have switched from pyrsistent to a Rust-based package.
We use the newest versions that don't pull in the build dependency on the Rust compiler.

• Resolves Make jupyterlab a standard package, update notebook to 7.x #24904
• Resolves Meta-ticket: Transition to JupyterLab #30399
• Resolves Update ipykernel to 6.13.0 - reintroduces psutil #33772

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

• Depends on Upgrade ipython to 8.17.x #35251 (merged here)

[mkoeppe]

sage -t --long --random-seed=286735480429121101562228604801325644303 sage/interacts/test_jupyter.rst
**********************************************************************
File "sage/interacts/test_jupyter.rst", line 9, in sage.interacts.test_jupyter
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interacts.test_jupyter[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/interacts/test_jupyter.rst", line 10, in sage.interacts.test_jupyter
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interacts.test_jupyter[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/interact.py
**********************************************************************
File "sage/repl/ipython_kernel/interact.py", line 12, in sage.repl.ipython_kernel.interact
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.interact[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/repl/ipython_kernel/interact.py", line 13, in sage.repl.ipython_kernel.interact
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.interact[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/widgets.py
**********************************************************************
File "sage/repl/ipython_kernel/widgets.py", line 11, in sage.repl.ipython_kernel.widgets
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/repl/ipython_kernel/widgets.py", line 12, in sage.repl.ipython_kernel.widgets
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
1 item had failures:
   2 of   3 in sage.repl.ipython_kernel.widgets
    [99 tests, 2 failures, 0.48 s]
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/kernel.py
**********************************************************************
File "sage/repl/ipython_kernel/kernel.py", line 48, in sage.repl.ipython_kernel.kernel.SageKernel.__init__
Failed example:
    from sage.repl.ipython_kernel.kernel import SageKernel
Expected nothing
Got:
    0.00s - Debugger warning: It seems that frozen modules are being used, which may
    0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
    0.00s - to python to disable frozen modules.
    0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
**********************************************************************
1 item had failures:
   1 of   3 in sage.repl.ipython_kernel.kernel.SageKernel.__init__
    [12 tests, 1 failure, 0.93 s]
sage -t --long --random-seed=286735480429121101562228604801325644303 sage/repl/ipython_kernel/widgets_sagenb.py
**********************************************************************
File "sage/repl/ipython_kernel/widgets_sagenb.py", line 11, in sage.repl.ipython_kernel.widgets_sagenb
Failed example:
    from ipywidgets.widgets.tests.utils import setup_test_comm
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets_sagenb[0]>", line 1, in <module>
        from ipywidgets.widgets.tests.utils import setup_test_comm
      File "/sage/local/var/lib/sage/venv-python3.11.1/lib/python3.11/site-packages/ipywidgets/widgets/tests/utils.py", line 50, in <module>
        orig_comm = ipykernel.comm.comm.BaseComm
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: module 'ipykernel.comm.comm' has no attribute 'BaseComm'
**********************************************************************
File "sage/repl/ipython_kernel/widgets_sagenb.py", line 12, in sage.repl.ipython_kernel.widgets_sagenb
Failed example:
    setup_test_comm()
Exception raised:
    Traceback (most recent call last):
      File "/sage/src/sage/doctest/forker.py", line 709, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/sage/src/sage/doctest/forker.py", line 1144, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.repl.ipython_kernel.widgets_sagenb[1]>", line 1, in <module>
        setup_test_comm()
        ^^^^^^^^^^^^^^^
    NameError: name 'setup_test_comm' is not defined
**********************************************************************
1 item had failures:
   2 of   5 in sage.repl.ipython_kernel.widgets_sagenb

[egourgoulhon]

Thank you for this update!

Built successfully with system Python (3.10.12) on Ubuntu 22.04 after pulling this PR branch into a fresh git clone of sage/develop.

• got the same doctest errors as Notebook 7, ipykernel 6.27, ipython 8.17 #36129 (comment) above.
• ran the Jupyter notebook with sage -n successfully on the notebook https://github.com/egourgoulhon/SageMathTest/blob/master/Notebooks/test_display.ipynb : this tests various displays: LaTeX, interactive widgets, 2d and 3d plots, animated 3d plots, Matplotlib interface, etc., especially in the scope of %display latex. The Matplotlib interface (activated via %matplotlib widget) required the package ipympl, which had to be installed by ./sage -pip install ipympl. Could ipympl be made a standard package, so that the interactive Matplotlib interface works out of the box?
• when running the same notebook within Jupyterlab (via sage -n and a click on the the Jupyterlab button), the LaTeX displays turned out to be ugly, while they are fine with the Jupyter notebook interface.

[mkoeppe]

Thanks for testing!

[mkoeppe]

Could ipympl be made a standard package, so that the interactive Matplotlib interface works out of the box?

Done in 26a4773

[mkoeppe]

• when running the same notebook within Jupyterlab (via sage -n and a click on the the Jupyterlab button), the LaTeX displays turned out to be ugly, while they are fine with the Jupyter notebook interface.

Could you check if installing https://pypi.org/project/jupyterlab-mathjax3/ makes a difference?

[egourgoulhon]

• when running the same notebook within Jupyterlab (via sage -n and a click on the the Jupyterlab button), the LaTeX displays turned out to be ugly, while they are fine with the Jupyter notebook interface.

Could you check if installing https://pypi.org/project/jupyterlab-mathjax3/ makes a difference?

Actually, this turned out to be not necessary: if one launches Jupyterlab via sage -n jupyterlab the LaTeX displays are OK. They are only ugly (weird fonts) when one launches Jupyterlab by cliking on the JupyterLab button in the top right menu of a notebook opened in the classic notebook mode, i.e. launched with sage -n. Since sage -n jupyterlab is the standard way (as currently documented) to run Sage with Jupyterlab at this stage, the issue is thus quite minor.

[egourgoulhon]

Thank you for making ipympl standard! I've just checked it in a brand new install of this PR branch merged in Sage 10.2.beta0. This indeed makes the interactive Matplolib window work out of the box in Sage's Jupyterlab.

[orlitzky]

File "sage/repl/ipython_kernel/kernel.py", line 48, in sage.repl.ipython_kernel.kernel.SageKernel.__init__
Failed example:
    from sage.repl.ipython_kernel.kernel import SageKernel
Expected nothing
Got:
    0.00s - Debugger warning: It seems that frozen modules are being used, which may
    0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
    0.00s - to python to disable frozen modules.
    0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

That's one of the failures I have from #36141

Adding PYDEVD_DISABLE_FILE_VALIDATION=1 to sage-env would be an easy solution unless someone needs to add breakpoints to a frozen python core module. It's not unthinkable, but the alternative to forbidding it (disabling frozen modules whenever we might import the sage kernel) seems worse.

What do you think? I can still push a fix to #36141.

[mkoeppe]

Is this coming from debugpy?

[orlitzky]

Is this coming from debugpy?

debugpy's bundled pydevd to be exact. The warning is raised when you import debugpy, like ipykernel does:

sage: from debugpy.server import api
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

It's a new warning but the behavior hasn't actually changed. If you were already using python-3.11, you can't put breakpoints in frozen modules. Now you still can't, and you get a warning about it at an irrelevant time. Setting PYDEVD_DISABLE_FILE_VALIDATION=1 essentially puts you back where you were: no breakpoints no warning.

[egourgoulhon]

I've just noticed an issue with the Notebook 7, both with jupyterlab and the classic notebook: in %display latex mode, long outputs are truncated, with no scrolling bar:

while in Sage 10.1 (running the Notebook 6) the output is fully displayed on several lines:

[dimpase]

to me Rust sounds less insane than Node.js. Anyway, can we use binary wheels for a package which needs Rust to build from source?

[mkoeppe]

can we use binary wheels

so far we don't have a mechanism for this for standard packages

[dimpase]

can we use binary wheels

so far we don't have a mechanism for this for standard packages

This is something that looks very meaningful to have. Is it somewhere in a written roadmap?

a workaround for this would be to also allow packages installed by the "building" user, via
pip install ... --user, or use --enable-system-site-packages from #36256 in a venv (e.g. pyenv).

[dimpase]

exceptiongroup is used in ipython docs, as well as in it itself (only for Python3 version < 3.11)
cf. https://github.com/ipython/ipython/blob/dd7f1e1894395f59649a84f94af8c9710202b08f/setup.cfg#L34

[mkoeppe]

Apparently this is another backport package that we need to add. exceptiongroup is a new feature in Python 3.11.
https://pypi.org/project/exceptiongroup/

[mkoeppe]

Tested successfully with ./configure --with-python=python3.9

[orlitzky]

Apparently this is another backport package that we need to add. exceptiongroup is a new feature in Python 3.11. https://pypi.org/project/exceptiongroup/

Can you add an spkg-configure.m4 to skip it for >=python-3.11 pleeeeeease? A copy/paste of #36776 should work.

[orlitzky]

I don't think we should mention it here unless we plan to make this a dependency of sagelib rather than something that just incidentally works?

[mkoeppe]

Hm... I agree that we should mention it there if and only if we make it a sagelib dependency.

I could go with either way -- adding such a backport package in general enables developers to use Python 3.11 features earlier, which is a good thing.

But I'll take out this commit here; we can consider it on another PR.

[mkoeppe]

Can you add an spkg-configure.m4 to skip it for >=python-3.11 pleeeeeease? A copy/paste of #36776 should work.

I'd rather not do it here. By the way, I think your solution in #36776 is not complete, as it seems to disregard the case --without-system-python3

[github-actions]

Documentation preview for this PR (built with commit 744e4b7; changes) is ready! 🎉

[egourgoulhon]

Thanks for having added the exceptiongroup package. Sage, as built with Ubuntu 22.04's default python (3.10.12), can now be launched without any error. I've tested both the (new Jupyterlab-based) Jupyter notebook and Jupyterlab on these two notebooks: test_display.ipynb and first_contact.ipynb.
Everything works nicely, except for two minor points in test_display.ipynb:

• in cell 18, the Ipython-native LaTeX display fails in the framework of %display latex
• in cell 21, which invokes %matplotlib widget, a deprecation warning shows up:

/home/eric/sage/10.2-test/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/traitlets/traitlets.py:1387: DeprecationWarning: Passing unrecognized arguments to super(Toolbar).__init__().
NavigationToolbar2WebAgg.__init__() missing 1 required positional argument: 'canvas'
This is deprecated in traitlets 4.2.This error will be raised in a future release of traitlets.

None of these issues is related to this PR, so I guess they can be fixed in a subsequent PR.

Side note: it is very nice to have Jupyterlab working out of the box, moreover in a recent version. Thank you Matthias!

[mkoeppe]

Thanks a lot for testing!

[egourgoulhon]

Apparently, with the new setting, the latex display does not work when there is no internet connection; see #36914.