-
-
Notifications
You must be signed in to change notification settings - Fork 404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sage should upgrade to jupyterlab 4.1.3 and notebook 7.1.1 for 10.3 #37533
Comments
I tested updating those two spkgs and it was not sufficient to fix the documentation issue. I think that some other dependencies also need to be updated. I will test further. |
It seems that those versions of the packages have new dependencies: httpx, h11, and httpcore. |
I presume you can update this with just By the way, there are new deps, how about ones which are no longer needed? That's why we need standard pip packages - in order not to do |
Yes, of course. And that will install httpx, h11 and httpcore in my .sage directory along with the newer notebook and jupyterlab packages. But that is not actually relevant for me. I am building the macOS binary package, so these things need to be installed in the sage venv, not in my .sage directory. My goal is that the app should work "out of the box", rather than on;y after the user figures out how to install some mysterious list of packages that are supposed to be included. I am still working on this. I added spkgs for those three packages and that makes the documentation links appear in the help menu. So far so good. But running the notebook server generates tracebacks and the shut down option does not appear in the File menu, with the result that server processes never die if started from the app. I think some other packages need to be updated to newer versions. |
You can tell pip where to install things. Actually, unless it had changed recently, You can create a venv, It doesn't help much with what packages are obsolete now, as it's all a fine mess. There is a helpful tool called |
it would be great to get rid of "I did it my way" sage venv, and allow Sage to be installable into a ready standard venv. |
It looks like jsonpointer is required by jsonschema which is required by jupyterlab_server. The macOS app installs packages in ~/.sage because installing into the app itself breaks the signature. At this point all versions are within the specified ranges, but that doesn't guarantee that they work correctly. Only that the work to some extent. The notebook works, despite the tracebacks. But with a clean install there are unhandled exceptions. |
at least according to
shown at top level, and only once; thus, not needed. (Or you can see this from Same applies to zipp, tomli, tzlocal, uri-template. (It can be a weird non-Python dependency, potentially: e.g. a JS package such as jupyter(lab)
|
Aren't we talking about building a venv to be packaged as macOS app, rather than what the app will do? |
oops, it's a optional dependency of jsonschema. Declared in |
Yes, see the PR that I prepared - #37535 |
After extensive testing I found that #37535 is not sufficient to allow me to remove the additional venv from the macOS app. I think it does make the documentation links appear, however. So it is definitely an improvement and I support the pull request. Sage is providing a more or less random collection of notebook and jupyterlab dependencies, with version numbers that differ significantly from those that one gets by installing those two packages in a fresh venv. What happens with Sage's collection of dependencies is that the notebook no longer provides a menu option to shut down the server, despite the fact that such an option is provided by default according to the settings. So this is a subtle, minor jupyter bug. It is important for the macOS Sage app, though, because when you launch a notebook from an app you do not have a terminal window on your screen which is full of obscure messages and ending with the statement that you should hit Control-C twice to kill the server. In fact there is no obvious way to kill the server other than by opening a terminal, searching for the pid and killing it. Of course the jupyter project cannot be expected to test every combination of versions for their dependencies which satisfy their stated requirements. Nor can they be expected to debug subtle issues that only arise with one particular collection of such dependencies. So I see no way to get this fixed. A clean venv with an installation of the jupyterlab and notebook packages basically provides a small list of standalone executables, the important ones being jupyter-notebook and jupyter-lab. Those executables are completely independent from Sage's kernel. However, providing those two executables requires 193 different python packages. Sage somehow is managing to provide all 193 packages with versions in the published required ranges. I would bet that many of those packages (but certainly not all since ipython is one of them) are only used by the notebooks. But figuring out the actual dependencies is a daunting prospect. With the additional venv that I am currently embedding in the app, all I need to do to get up-to-date notebook servers is to make a small change to the sage-notebook script. Basically all I do is change the shebang so that it runs the jupyter-notebook or jupyter-lab program in the extra venv instead of the one in the sage venv. While that setup makes for a fair amount of redundancy it has the virtue of being simple. |
Are you saying that you are not using Sage's jupyterlab in your app, but, basically, a standalone jupyterlab? |
Yes. I considered trying to remove the jupyter packages from Sage to save some space. But I was not confident that I could do that without breaking things, especially ipython. By the way, I was confused by the jupyter_notebook_config.py file in build/pkgs/notebook. I don't know what it is doing there but I think it is not used by anything. |
so much for @NathanDunfield saying that the current 193+ vendored jupyter-related packages provide more "stability" than simply letting us use jupyterlab as a pip package 🥲 |
That is not what Nathan said. You apparently missed his entire point. |
Before the update to Notebook 7, we used to copy it (in I don't know if anything in it is still valuable. |
We have |
he said: "I think the current system of having everything pinned and explicitly recorded is the right choice, being more stable in my experience with other projects. " I presume it was an argument in favour of 193+ random package versions "pinned and explicitly recorded" ? Aren't we seeing here that it's not really right choice? |
I have just discovered that the menu item for shutting down the server is missing with Sage-10.3.rc2 whether or not I use the separate venv. It does appear in the pre-release version of the app that used Sage-10.3.rc1. I don't know what changed between those releases that could have affected this, but very little changed in Sage. So I think the change must have been in jupyter. |
I've just installed the latest jupyterlab (4.1.3) in a venv, and, sure enough, in File menu there is a Shut Down item, right at the bottom of the menu. It shuts down the jupyterlab server (and closes the browser window) just fine. Tested with Chrome, ymmv. So perhaps with 4.1.2 it went AWOL, but it's back now. Could you update the issue title to say 4.1.3, maybe? |
Yes, it was: when you have a dependency, pinning the version increases stability.
In this case, a feature was added/a bug was fixed in 7.1.1 compared to the pinned version, but with notebook 7.1.2 the reverse could easily be true. Or one of notebooks's many dependencies could release a new version, which turns out to break the latest version of notebook. Pinning prevents regressions, but of course one cost is that one doesn't automatically gain any improvements. That's a tradeoff I'm in favor of making, especially for a GUI component like JupyterLab that's hard to test automatically. |
I think that the "shut down" menu got lost in the upgrade of jupyter-server from 2.12.5 to 2.13.0. But I am still working on verifying that. |
As far as I know, there is no notebook 7.1.2 yet, there is 7.1.1, see https://pypi.org/project/notebook/ |
with jupterlab 4.1.3, it doesn't matter whether I have jupyter-servers 2.12.5 or 2.13.0, I have this menu item working. |
Tradeoff ? Very many users run notebook/jupyter(lab) with unpinned dependencies, so many more eyes on GUI compared to a setup with fixed to rather random values versions - and probably some projects even have automated testing of GUI features - that's much better than reports from a tiny, in comparison, userbase of Sage. I'd say that for a release like macOS app, the normal scheme would be to pin versions to whatever values one gets from |
Except for one key fact. It turns out that it is not a better way, and this story is a case in point -- demonstrating that, as usual, Nathan was right and we would all do well to listen to him. When I replaced jupyter-server 2.13.0 (released on March 4) with the jupyter-server 2.12.5 that I had used in the pre-release version of the app the "Shut Down" menu came back. So this feature was broken on March 4. After that day it was no longer the case that installing notebook and jupyterlab in a clean venv worked correctly. (Maybe, in honor of Pogo, we should call this bug the March Forth! bug.) Moving forward, I am going to pin all of the versions that get installed in the notebook venv in the macOS app. This is easy to do with a requirements.txt file, although that file has to be maintained. Still, maintaining a 193 line requirements.txt file is a lot easier than maintaining 193 spkg directories. Incidentally, spkg directories for wheel packages contain almost no information which is not available from Pypi via their REST API. The exception is the SPKG.rst file, which is used by the documentation and apparently must be written by hand, although it need not necessarily change with a simple version update. Yesterday I wrote a python package to generate spkg directories from data obtained from the PyPI API. If anyone is interested in using such a thing to simplify Sage maintenance, please contact me. |
That's true with the current tooling. A simple script would close this gap: |
it's already working with --enable-system-side-packages |
you are packaging a venv - it's sage's own venv. one should make sage installable in an existing venv, as normal python packages do. |
Of course, but there is a lot more than that which goes into the package. The main rules, as far as making a macOS App bundle notarizable, are:
Other than that you can put most anything you want, including a venv, into the bundle. But, in the case of a venv, the python installation referenced by the venv must reside inside the bundle. So it is essential that we use Sage's python, not some random python installed on the system. These days, in the so-called python ecosystem, most binary wheels that need dynamic libraries use the delocate package to embed those libraries into the python package inside of a subdirectory named .dylibs. We have to be careful to find and sign those. For Sage packages that use dynamic libraries we need to rewrite rpaths to make them relative to the extension module (.so) file. |
In the user's jail misnamed macOS, one has to install as many copies of Python as one has Python projects? Wonderful - so Apple can sell you a new computer every year, as you keep running out of disk space :-) |
Essential? Surely there are tools to clone enough Python into a venv so that it becomes suitable for an app, cf. e.g. py2app |
Buiding python is trivial. "Cloning" some random python would not be. Plus it does not go into a venv. Please let me worry about this - I have a lot of experience with it by now and I know where the difficult points are. Compiling python is not one of them. If you want to do it your way, please feel free. |
I am merely talking about a normal way people do apps with Python. You don't need a bundled with Sage Python sources for this. |
Yes, all macOS apps based on python do bundle some version of python. Sometimes it is copied from the python.org installation. Other times it is not . There can be issues with doing it. For example, py2app, which is one of the most common ways of doing that, still does not support 3.12 after 6 months, due to changes to importlib. We have apps using 3.12, and we do it by building our own python, which is easy. There is no way that py2app could possibly cope with the size and complexity of Sage. As long as the sage build works, which it does with never an issue, it is simplest to use it. Honestly, there are some serious problems with building this app but compiling python is not one of them. The python developers are careful and produce an excellent product which builds easily in an amazing range of contexts. I am not going to completely rework my build process to solve a problem which does not exist. |
Vendoring something without patching it is seldom justified. |
I have certainly heard that story before. But it doesn't make sense. If you are not patching anything then why is it such a big burden? And, if it is such a big burden, why did you just suggest "cloning" python? This discussion is a dead end and completely irrelevant to this issue. I am finished with it. |
it doesn't make sense to juggle different versions of python yet demand that Sage keeps vendoring a python for you. besides, even unpatched vendored dependencies are a hassle, as you learnt in the case of Jupyter/notebook. it is also a hassle because if something is not right with, say, Jupyter, or, say, ssl support in python, you bother us, and not the upstream, which knows much more about any potential problems with their soft, gets much more feedback, has more resources to help you, etc. |
We do not "juggle" versions of python. We ship the python built by sage. I asked you to end this discussion which is not related in any way to this issue. Why are you still at it? |
Like I said in https://groups.google.com/g/sage-devel/c/rBvXdVKC4nw/m/Zt0dykgbAgAJ, I consider @culler's efforts to provide a self-contained macOS app of Sage to be a very valuable service for the Sage project. |
yes, you ship Python 3.11.1, with quite a few known and fixed in the latest 3.11, i.e. 3.11.8, CVEs and bugs. You should not trust Sage to give you up to date things among the gazzillion of vendored packages. |
I don't question its value, I would like to consider what their experiences tell the Sage project. We see they ignore Sage's Jupyter. I'm trying to see what prevents them from ignoring Sage's Python (not much, it seems). |
If you have chance, try if #37247 improves this. It adds some packages to the list of those are disabled with
I don't think it does that. The mechanism is really very simple: https://github.com/sagemath/sage/blob/develop/configure.ac#L418 |
I tried again, using a clean clone of sage. When I configure with --disable-notebook the configure script reports that the notebook packages listed on that line in configure.ac have been disabled by a configure option. But then when I run make build it goes ahead and installs all of those packages anyway. |
sphinx uses Jupyter, so you need |
The only change in #37247 that seemed relevant was the line in configure.ac that you mentioned, which adds a couple of packages to the list of packages to disable when --disable-notebook is set. I added even more packages to that line. When I ran configure it reported that all of the listed packages had been disabled and would not be built. Then those packages were built and installed anyway. I guess I can just pip-uninstall them, although that is a waste of time. Now that I have figured out how to keep sage from rebuilding gmp and everything that depends on it every run, I have time to spare. |
For the macOS Sagemath-10.3 release I am now building Sage with --disable-notebook (which currently has no effect, but maybe will work someday). After the build I run:
where the requirements.txt file was generated by pip freeze in a venv where I had installed jupyterlab, notebook and pillow[1]. This installs 90 packages from binary wheels (not 193, which was a mistake involving counting dist-info directories as packages), most of which replace packages built (needlessly) by Sage from sdists. Doing this allows me to remove the redundant secondary venv that I used for SageMath-10.2, reducing the size of the final disk image by 100MB, back to 1.0 GB for the Intel build. And it means that the jupyterlab and notebook packages are fully up-to-date [2] for the macOS binary package. The requirements.txt file will be part of the 3-manifolds/Sage_macOS repository and will serve as the means of pinning versions going forward. Everything seems to be working, with the exception of some intermittent failures with the ipywidgets interact function / decorator. That seems to usually work reliably in jupyterlab, but often fails in notebook, but will start working in notebook if you switch to jupyterlab and back. When I did manage to make it fail in jupyterlab the jupyterlab log console (a feature which does not exist in notebook as far as I can tell) reports that a "comm" identified by some hash value does not exist. Debugging minimized obfuscated javascript code generated by the Typescript compiler is not something I am capable of, so I think I have to live with this flakiness for the moment. (I would be happy to post an issue at jupyter, but I am not able to reproduce this failure when Sage is not involved, so I don't think they would be willing to work on it.) [1] Using a binary wheel for pillow allows me to bypass the issue of compiling openjpeg and provides all of the image formats that pillow supports. [2] Without a change in Sage policies it is not possible to distribute an up-to-date version of jupyter-server with Sage. The reason is that the jupyter-server package depends on pyrpds, which is a python wrapper for a Rust library. Under the current policies, pyrpds would have to be built from an sdist, which would require embedding a Rust toolchain within the Sage distribution. If one is connected to the internet it is an easy one-liner to install a Rust toolchain in one's home directory, but I would have no idea how to go about embedding such a thing in Sage, and my impression is that the Sage community would not be willing to consider doing that anyway. I think that it would be wise for Sage to consider the possibility of having a new type of spkg which is installed from a binary wheel. I believe that would be a simple change, if people were willing to consider it, involving just adding a new "Sage distribution helper". |
The "new type of package" here is just a standard pip package (not allowed under the current policy). As you know, I have proposed to lift this restriction on sage-devel. |
Sage's "standard pip packages", as currently defined, must use "none-any" wheels and must be built from an sdist unless they are pure python packages. This new type would use a platform-specific binary wheel containing shared libraries (presumably manylinux in the case of linux systems). Perhaps you were using the term "standard pip package" to mean the sort of package that one gets by just running "pip install". But Sage has its own clear definition of the term, so using the same term with a different meaning is likely to be confusing. I don't know who, if anyone, understood that you were talking about using platform-specific binary wheels in your proposal. |
I don't think it is likely to be productive to discuss something like this in the abstract, with the whole group. To make it happen I think it would be necessary to fork Sage and implement the needed changes and demonstrate that it works. If that were done it would be a concrete proposal and it would be very clear what is being proposed. I don't think it would require any major changes, either. |
There is one sort of major thing that would need to change: namely the documentation requirements. Matthias would like to expand the documentation to explain how each package fits into Sage. Realistically, though, with a package like pyrpds, the best we could hope to do is say "we need this because we provide jupyter notebooks and they require it". |
You are right. I keep thinking of the latter, as my main examples of packages that ought to be like this are pure python packages.
My proposal is about a possibility to have a pip standard package. TBF, I didn't think at all about this technicality, I'd be happy to just have pure Python standard pip packages. But, indeed, rust is getting more and more popular in Python world, and more and more packages would use it, making them rather unpleasant to ship as sources (the last thing Sage needs it an extra stress of packaging a rust toolchain) |
Why? There are clearly packages which are ripe for becoming pip standard packages, e.g. pytest. Given that Sage is the only known to me open-source Python project packaging compiler toolchains and Jupyter, I really doubt one needs a prototype. Besides, prototypes exist in few Linux distros: Gentoo, Arch, Void (and in Conda) - prototypes which only use the core sagelib, whereas the rest comes straight from the OS (or conda binary installs, in Conda case). |
Problem Description
The versions of jupyterlab and notebook used by the 10.3rc1 release (4.0.6 and 7.0.6) have a problem which is specifically relevant to Sage and which is fixed in the slightly newer versions suggested here. The problem is that the Sage documentation links do not appear in the Help menu in the notebook. I opened a ticket for this with the notebook project at the time of the 10.2 release. They fixed it, but the the fix did not take effect immediately because it depended on a change to Jupyterlab which had not been released at that time. The fix is now in place with jupyterlab 4.1.2 and notebook 7.1.1. I have tested these (they are included in the Sage_macOS 10.3 prerelease) and the links do appear correctly in the Help menu, and they work.
Proposed Solution
Make these small bumps (4.0.6 -> 4.1.2 and 7.0.6 -> 7.1.1) in the package versions used in these two wheel spkgs.
Alternatives Considered
Continuing to use a redundant second venv in the macOS binary.
Additional Information
Currently the Sage_macOS binary includes a second, mostly redundant, venv in which these packages and their dependencies are installed. This ends up being fairly large, because there are lots of dependencies. I think the redundant venv could be removed if Sage made this minor upgrade. This would be a significant reduction (176 MB) in the size of the binary package.
Is there an existing issue for this?
The text was updated successfully, but these errors were encountered: