Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sage should upgrade to jupyterlab 4.1.3 and notebook 7.1.1 for 10.3 #37533

Closed
1 task done
culler opened this issue Mar 3, 2024 · 73 comments · Fixed by #37535
Closed
1 task done

Sage should upgrade to jupyterlab 4.1.3 and notebook 7.1.1 for 10.3 #37533

culler opened this issue Mar 3, 2024 · 73 comments · Fixed by #37535

Comments

@culler
Copy link
Contributor

culler commented Mar 3, 2024

Problem Description

The versions of jupyterlab and notebook used by the 10.3rc1 release (4.0.6 and 7.0.6) have a problem which is specifically relevant to Sage and which is fixed in the slightly newer versions suggested here. The problem is that the Sage documentation links do not appear in the Help menu in the notebook. I opened a ticket for this with the notebook project at the time of the 10.2 release. They fixed it, but the the fix did not take effect immediately because it depended on a change to Jupyterlab which had not been released at that time. The fix is now in place with jupyterlab 4.1.2 and notebook 7.1.1. I have tested these (they are included in the Sage_macOS 10.3 prerelease) and the links do appear correctly in the Help menu, and they work.

Proposed Solution

Make these small bumps (4.0.6 -> 4.1.2 and 7.0.6 -> 7.1.1) in the package versions used in these two wheel spkgs.

Alternatives Considered

Continuing to use a redundant second venv in the macOS binary.

Additional Information

Currently the Sage_macOS binary includes a second, mostly redundant, venv in which these packages and their dependencies are installed. This ends up being fairly large, because there are lots of dependencies. I think the redundant venv could be removed if Sage made this minor upgrade. This would be a significant reduction (176 MB) in the size of the binary package.

Is there an existing issue for this?

  • I have searched the existing issues for a bug report that matches the one I want to file, without success.
@culler
Copy link
Contributor Author

culler commented Mar 3, 2024

I tested updating those two spkgs and it was not sufficient to fix the documentation issue. I think that some other dependencies also need to be updated. I will test further.

@culler
Copy link
Contributor Author

culler commented Mar 4, 2024

It seems that those versions of the packages have new dependencies: httpx, h11, and httpcore.

@dimpase
Copy link
Member

dimpase commented Mar 4, 2024

I presume you can update this with just ./sage --pip install notebook -U, no?

By the way, there are new deps, how about ones which are no longer needed?
E.g. jsonpointer ?


That's why we need standard pip packages - in order not to do pip's work ourselves all the time.

@culler
Copy link
Contributor Author

culler commented Mar 4, 2024

Yes, of course. And that will install httpx, h11 and httpcore in my .sage directory along with the newer notebook and jupyterlab packages. But that is not actually relevant for me. I am building the macOS binary package, so these things need to be installed in the sage venv, not in my .sage directory. My goal is that the app should work "out of the box", rather than on;y after the user figures out how to install some mysterious list of packages that are supposed to be included.

I am still working on this. I added spkgs for those three packages and that makes the documentation links appear in the help menu. So far so good. But running the notebook server generates tracebacks and the shut down option does not appear in the File menu, with the result that server processes never die if started from the app. I think some other packages need to be updated to newer versions.

@dimpase
Copy link
Member

dimpase commented Mar 4, 2024

You can tell pip where to install things. Actually, unless it had changed recently, ./sage --pip install used to install into Sage packages tree, not to ~/.sage

You can create a venv, pip install notebook there, and take a note of packages+versions needed.

It doesn't help much with what packages are obsolete now, as it's all a fine mess.

There is a helpful tool called pipdeptree (pip-installable) which allows you to display python packages nicely, as a proper forest,
or actually create a requirements.txt file (which may list top-level packages only)

@dimpase
Copy link
Member

dimpase commented Mar 4, 2024

it would be great to get rid of "I did it my way" sage venv, and allow Sage to be installable into a ready standard venv.

@culler
Copy link
Contributor Author

culler commented Mar 4, 2024

It looks like jsonpointer is required by jsonschema which is required by jupyterlab_server.

The macOS app installs packages in ~/.sage because installing into the app itself breaks the signature.

At this point all versions are within the specified ranges, but that doesn't guarantee that they work correctly. Only that the work to some extent. The notebook works, despite the tracebacks. But with a clean install there are unhandled exceptions.

@dimpase
Copy link
Member

dimpase commented Mar 4, 2024

at least according to pipdeptree, jsonpointer is no longer needed; on a venv with pip-installed notebook and sphinx*-stuff:

$ pipdeptree | grep jsonpointer
jsonpointer==2.4

shown at top level, and only once; thus, not needed. (Or you can see this from pip show jsonpointer)

Same applies to zipp, tomli, tzlocal, uri-template.

(It can be a weird non-Python dependency, potentially: e.g. a JS package such as jupyter(lab)
invokes one of these Python packages in a weird way, but I doubt - this would be an awful design)

├── jupyterlab_server [required: >=2.22.1,<3, installed: 2.25.3]
│   ├── Babel [required: >=2.10, installed: 2.14.0]
│   ├── Jinja2 [required: >=3.0.3, installed: 3.1.3]
│   │   └── MarkupSafe [required: >=2.0, installed: 2.1.5]
│   ├── json5 [required: >=0.9.0, installed: 0.9.17]
│   ├── jsonschema [required: >=4.18.0, installed: 4.21.1]
│   │   ├── attrs [required: >=22.2.0, installed: 23.2.0]
│   │   ├── jsonschema-specifications [required: >=2023.03.6, installed: 2023.12.1]
│   │   │   └── referencing [required: >=0.31.0, installed: 0.33.0]
│   │   │       ├── attrs [required: >=22.2.0, installed: 23.2.0]
│   │   │       └── rpds-py [required: >=0.7.0, installed: 0.18.0]
│   │   ├── referencing [required: >=0.28.4, installed: 0.33.0]
│   │   │   ├── attrs [required: >=22.2.0, installed: 23.2.0]
│   │   │   └── rpds-py [required: >=0.7.0, installed: 0.18.0]
│   │   └── rpds-py [required: >=0.7.1, installed: 0.18.0]

@dimpase
Copy link
Member

dimpase commented Mar 4, 2024

The macOS app installs packages in ~/.sage because installing into the app itself breaks the signature.

Aren't we talking about building a venv to be packaged as macOS app, rather than what the app will do?

@dimpase
Copy link
Member

dimpase commented Mar 4, 2024

jsonpointer

oops, it's a optional dependency of jsonschema. Declared in [project.optional-dependencies] list of jsonschema's pyproject.toml. Apparently pip isn't smart enough to show them easily.

@mkoeppe
Copy link
Member

mkoeppe commented Mar 5, 2024

It seems that those versions of the packages have new dependencies: httpx, h11, and httpcore.

Yes, see the PR that I prepared - #37535

@culler
Copy link
Contributor Author

culler commented Mar 6, 2024

After extensive testing I found that #37535 is not sufficient to allow me to remove the additional venv from the macOS app. I think it does make the documentation links appear, however. So it is definitely an improvement and I support the pull request.

Sage is providing a more or less random collection of notebook and jupyterlab dependencies, with version numbers that differ significantly from those that one gets by installing those two packages in a fresh venv. What happens with Sage's collection of dependencies is that the notebook no longer provides a menu option to shut down the server, despite the fact that such an option is provided by default according to the settings. So this is a subtle, minor jupyter bug. It is important for the macOS Sage app, though, because when you launch a notebook from an app you do not have a terminal window on your screen which is full of obscure messages and ending with the statement that you should hit Control-C twice to kill the server. In fact there is no obvious way to kill the server other than by opening a terminal, searching for the pid and killing it.

Of course the jupyter project cannot be expected to test every combination of versions for their dependencies which satisfy their stated requirements. Nor can they be expected to debug subtle issues that only arise with one particular collection of such dependencies. So I see no way to get this fixed.

A clean venv with an installation of the jupyterlab and notebook packages basically provides a small list of standalone executables, the important ones being jupyter-notebook and jupyter-lab. Those executables are completely independent from Sage's kernel. However, providing those two executables requires 193 different python packages. Sage somehow is managing to provide all 193 packages with versions in the published required ranges. I would bet that many of those packages (but certainly not all since ipython is one of them) are only used by the notebooks. But figuring out the actual dependencies is a daunting prospect.

With the additional venv that I am currently embedding in the app, all I need to do to get up-to-date notebook servers is to make a small change to the sage-notebook script. Basically all I do is change the shebang so that it runs the jupyter-notebook or jupyter-lab program in the extra venv instead of the one in the sage venv. While that setup makes for a fair amount of redundancy it has the virtue of being simple.

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

Are you saying that you are not using Sage's jupyterlab in your app, but, basically, a standalone jupyterlab?
I.e., if Sage only provided jupyter kernel, but no jupyter packages, it would still be fine for you?

@culler
Copy link
Contributor Author

culler commented Mar 6, 2024

Yes.

I considered trying to remove the jupyter packages from Sage to save some space. But I was not confident that I could do that without breaking things, especially ipython.

By the way, I was confused by the jupyter_notebook_config.py file in build/pkgs/notebook. I don't know what it is doing there but I think it is not used by anything.

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

so much for @NathanDunfield saying that the current 193+ vendored jupyter-related packages provide more "stability" than simply letting us use jupyterlab as a pip package 🥲

@culler
Copy link
Contributor Author

culler commented Mar 6, 2024

That is not what Nathan said. You apparently missed his entire point.

@mkoeppe
Copy link
Member

mkoeppe commented Mar 6, 2024

I was confused by the jupyter_notebook_config.py file in build/pkgs/notebook. I don't know what it is doing there but I think it is not used by anything.

Before the update to Notebook 7, we used to copy it (in build/pkgs/notebook/spkg-install.in) to SAGE_VENV/etc/jupyter/

I don't know if anything in it is still valuable.

@mkoeppe
Copy link
Member

mkoeppe commented Mar 6, 2024

I considered trying to remove the jupyter packages from Sage to save some space. But I was not confident that I could do that without breaking things, especially ipython.

We have ./configure --disable-notebook for this.

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

That is not what Nathan said. You apparently missed his entire point.

he said: "I think the current system of having everything pinned and explicitly recorded is the right choice, being more stable in my experience with other projects. "

I presume it was an argument in favour of 193+ random package versions "pinned and explicitly recorded" ?

Aren't we seeing here that it's not really right choice?

@culler
Copy link
Contributor Author

culler commented Mar 6, 2024

I have just discovered that the menu item for shutting down the server is missing with Sage-10.3.rc2 whether or not I use the separate venv. It does appear in the pre-release version of the app that used Sage-10.3.rc1. I don't know what changed between those releases that could have affected this, but very little changed in Sage. So I think the change must have been in jupyter.

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

I've just installed the latest jupyterlab (4.1.3) in a venv, and, sure enough, in File menu there is a Shut Down item, right at the bottom of the menu. It shuts down the jupyterlab server (and closes the browser window) just fine. Tested with Chrome, ymmv.

So perhaps with 4.1.2 it went AWOL, but it's back now. Could you update the issue title to say 4.1.3, maybe?

@NathanDunfield
Copy link

he said: "I think the current system of having everything pinned and explicitly recorded is the right choice, being more stable in my experience with other projects. "

I presume it was an argument in favour of 193+ random package versions "pinned and explicitly recorded" ?

Yes, it was: when you have a dependency, pinning the version increases stability.

Aren't we seeing here that it's not really right choice?

In this case, a feature was added/a bug was fixed in 7.1.1 compared to the pinned version, but with notebook 7.1.2 the reverse could easily be true. Or one of notebooks's many dependencies could release a new version, which turns out to break the latest version of notebook. Pinning prevents regressions, but of course one cost is that one doesn't automatically gain any improvements. That's a tradeoff I'm in favor of making, especially for a GUI component like JupyterLab that's hard to test automatically.

@culler
Copy link
Contributor Author

culler commented Mar 6, 2024

I think that the "shut down" menu got lost in the upgrade of jupyter-server from 2.12.5 to 2.13.0. But I am still working on verifying that.

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

As far as I know, there is no notebook 7.1.2 yet, there is 7.1.1, see https://pypi.org/project/notebook/

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

I think that the "shut down" menu got lost in the upgrade of jupyter-server from 2.12.5 to 2.13.0. But I am still working on verifying that.

with jupterlab 4.1.3, it doesn't matter whether I have jupyter-servers 2.12.5 or 2.13.0, I have this menu item working.

@dimpase
Copy link
Member

dimpase commented Mar 6, 2024

That's a tradeoff I'm in favor of making, especially for a GUI component like JupyterLab that's hard to test automatically.

Tradeoff ? Very many users run notebook/jupyter(lab) with unpinned dependencies, so many more eyes on GUI compared to a setup with fixed to rather random values versions - and probably some projects even have automated testing of GUI features - that's much better than reports from a tiny, in comparison, userbase of Sage.

I'd say that for a release like macOS app, the normal scheme would be to pin versions to whatever values one gets from pip install notebook in a clean venv.
And as macOS app finds this a better way, I don't see why we should do something different in general

@culler
Copy link
Contributor Author

culler commented Mar 6, 2024

Except for one key fact. It turns out that it is not a better way, and this story is a case in point -- demonstrating that, as usual, Nathan was right and we would all do well to listen to him.

When I replaced jupyter-server 2.13.0 (released on March 4) with the jupyter-server 2.12.5 that I had used in the pre-release version of the app the "Shut Down" menu came back. So this feature was broken on March 4. After that day it was no longer the case that installing notebook and jupyterlab in a clean venv worked correctly. (Maybe, in honor of Pogo, we should call this bug the March Forth! bug.)

Moving forward, I am going to pin all of the versions that get installed in the notebook venv in the macOS app. This is easy to do with a requirements.txt file, although that file has to be maintained. Still, maintaining a 193 line requirements.txt file is a lot easier than maintaining 193 spkg directories.

Incidentally, spkg directories for wheel packages contain almost no information which is not available from Pypi via their REST API. The exception is the SPKG.rst file, which is used by the documentation and apparently must be written by hand, although it need not necessarily change with a simple version update. Yesterday I wrote a python package to generate spkg directories from data obtained from the PyPI API. If anyone is interested in using such a thing to simplify Sage maintenance, please contact me.

@mkoeppe
Copy link
Member

mkoeppe commented Mar 6, 2024

maintaining a 193 line requirements.txt file is a lot easier than maintaining 193 spkg directories.

That's true with the current tooling. A simple script would close this gap:

@dimpase
Copy link
Member

dimpase commented Mar 7, 2024

Can one --disable-sphinx too?

You are looking for configure --disable-doc.

I could imagine a world in which Sage would use sphinx to build the html docs, but would not include sphinx inside of sage. That would probably be a delicate change to implement though.

it's already working with --enable-system-side-packages

@dimpase
Copy link
Member

dimpase commented Mar 7, 2024

--enable-system-site-packages used to work in an venv (which you can package just fine, no?), but seems to be broken now.

No, actually, I do not and cannot package a venv. I package the entire Sage build and delete stuff that is not useful. I have added an additional venv to the sage build and I add some other things outside of the sage build, including frameworks for Tcl and Tk, that make up the app. But that is not really relevant.

you are packaging a venv - it's sage's own venv.

one should make sage installable in an existing venv, as normal python packages do.

@culler
Copy link
Contributor Author

culler commented Mar 7, 2024

you are packaging a venv - it's sage's own venv.

Of course, but there is a lot more than that which goes into the package.

The main rules, as far as making a macOS App bundle notarizable, are:

  • it must not reference any dynamic libraries outside of the bundle, except for Apple's C runtime
    (This includes rpaths).
  • It must not contain any symlinks to files outside of the bundle
  • It must not have any broken symlinks
  • every exectuable file, and every framework, and the app itself must be signed.

Other than that you can put most anything you want, including a venv, into the bundle. But, in the case of a venv, the python installation referenced by the venv must reside inside the bundle. So it is essential that we use Sage's python, not some random python installed on the system.

These days, in the so-called python ecosystem, most binary wheels that need dynamic libraries use the delocate package to embed those libraries into the python package inside of a subdirectory named .dylibs. We have to be careful to find and sign those. For Sage packages that use dynamic libraries we need to rewrite rpaths to make them relative to the extension module (.so) file.

@dimpase
Copy link
Member

dimpase commented Mar 8, 2024

in the so-called python ecosystem

In the user's jail misnamed macOS, one has to install as many copies of Python as one has Python projects? Wonderful - so Apple can sell you a new computer every year, as you keep running out of disk space :-)

@dimpase
Copy link
Member

dimpase commented Mar 8, 2024

it is essential that we use Sage's python

Essential? Surely there are tools to clone enough Python into a venv so that it becomes suitable for an app, cf. e.g. py2app

@culler
Copy link
Contributor Author

culler commented Mar 8, 2024

Buiding python is trivial. "Cloning" some random python would not be. Plus it does not go into a venv.

Please let me worry about this - I have a lot of experience with it by now and I know where the difficult points are. Compiling python is not one of them. If you want to do it your way, please feel free.

@dimpase
Copy link
Member

dimpase commented Mar 8, 2024

I am merely talking about a normal way people do apps with Python. You don't need a bundled with Sage Python sources for this.

@culler
Copy link
Contributor Author

culler commented Mar 8, 2024

Yes, all macOS apps based on python do bundle some version of python. Sometimes it is copied from the python.org installation. Other times it is not . There can be issues with doing it. For example, py2app, which is one of the most common ways of doing that, still does not support 3.12 after 6 months, due to changes to importlib. We have apps using 3.12, and we do it by building our own python, which is easy. There is no way that py2app could possibly cope with the size and complexity of Sage.

As long as the sage build works, which it does with never an issue, it is simplest to use it. Honestly, there are some serious problems with building this app but compiling python is not one of them. The python developers are careful and produce an excellent product which builds easily in an amazing range of contexts. I am not going to completely rework my build process to solve a problem which does not exist.

@dimpase
Copy link
Member

dimpase commented Mar 8, 2024

Vendoring something without patching it is seldom justified.
And we have 250 or 300 such vendored packages.
It is a maintenance burden which gets heavier and heavier - but as you don't participate in shouldering it, you don't notice

@culler
Copy link
Contributor Author

culler commented Mar 8, 2024

I have certainly heard that story before. But it doesn't make sense. If you are not patching anything then why is it such a big burden? And, if it is such a big burden, why did you just suggest "cloning" python?

This discussion is a dead end and completely irrelevant to this issue. I am finished with it.

@dimpase
Copy link
Member

dimpase commented Mar 8, 2024

it doesn't make sense to juggle different versions of python yet demand that Sage keeps vendoring a python for you.

besides, even unpatched vendored dependencies are a hassle, as you learnt in the case of Jupyter/notebook.

it is also a hassle because if something is not right with, say, Jupyter, or, say, ssl support in python, you bother us, and not the upstream, which knows much more about any potential problems with their soft, gets much more feedback, has more resources to help you, etc.

@culler
Copy link
Contributor Author

culler commented Mar 8, 2024

We do not "juggle" versions of python. We ship the python built by sage.

I asked you to end this discussion which is not related in any way to this issue. Why are you still at it?

@mkoeppe
Copy link
Member

mkoeppe commented Mar 8, 2024

Like I said in https://groups.google.com/g/sage-devel/c/rBvXdVKC4nw/m/Zt0dykgbAgAJ, I consider @culler's efforts to provide a self-contained macOS app of Sage to be a very valuable service for the Sage project.
And the project has embraced it by advertising Marc's macOS app in the Download menu in https://www.sagemath.org/ and as the recommended method of installation on macOS in the Sage Installation Manual.

@dimpase
Copy link
Member

dimpase commented Mar 10, 2024

We ship the python built by sage.

yes, you ship Python 3.11.1, with quite a few known and fixed in the latest 3.11, i.e. 3.11.8, CVEs and bugs. You should not trust Sage to give you up to date things among the gazzillion of vendored packages.

@dimpase
Copy link
Member

dimpase commented Mar 10, 2024

I consider @culler's efforts to provide a self-contained macOS app of Sage to be a very valuable service for the Sage project

I don't question its value, I would like to consider what their experiences tell the Sage project. We see they ignore Sage's Jupyter. I'm trying to see what prevents them from ignoring Sage's Python (not much, it seems).

@mkoeppe
Copy link
Member

mkoeppe commented Mar 10, 2024

configure --disable-notebook should simply make make build not build the various notebook and jupyter packages. Does it not do that?

That is correct. It does not do that. It builds jupyter_client, jupyter_events, jupyter_server, jupyter_sphinx, jupyterlab_server, jupyterlab_widgets as well as packages that I think are only needed by jupyter such as tornado, anyio, argon2, fastjsonschema, sniffio etc.

If you have chance, try if #37247 improves this. It adds some packages to the list of those are disabled with --disable-notebook.

The logic seems to be that configure overrides the --disable-notebook flag unless --enable-system-site-packages is set to yes.

I don't think it does that. The mechanism is really very simple: https://github.com/sagemath/sage/blob/develop/configure.ac#L418

@mkoeppe mkoeppe added this to the sage-10.3 milestone Mar 10, 2024
@culler
Copy link
Contributor Author

culler commented Mar 11, 2024

I tried again, using a clean clone of sage. When I configure with --disable-notebook the configure script reports that the notebook packages listed on that line in configure.ac have been disabled by a configure option. But then when I run make build it goes ahead and installs all of those packages anyway.

@mkoeppe
Copy link
Member

mkoeppe commented Mar 11, 2024

@culler Is that with or without #37247?

@dimpase
Copy link
Member

dimpase commented Mar 11, 2024

sphinx uses Jupyter, so you need --disable-doc too, if you really want no traces of Jupiter in the installation

@culler
Copy link
Contributor Author

culler commented Mar 11, 2024

The only change in #37247 that seemed relevant was the line in configure.ac that you mentioned, which adds a couple of packages to the list of packages to disable when --disable-notebook is set.

I added even more packages to that line. When I ran configure it reported that all of the listed packages had been disabled and would not be built. Then those packages were built and installed anyway. I guess I can just pip-uninstall them, although that is a waste of time. Now that I have figured out how to keep sage from rebuilding gmp and everything that depends on it every run, I have time to spare.

@culler
Copy link
Contributor Author

culler commented Mar 12, 2024

For the macOS Sagemath-10.3 release I am now building Sage with --disable-notebook (which currently has no effect, but maybe will work someday). After the build I run:

venv/bin/python3 -m pip install --no-user --force-reinstall --upgrade-strategy eager -r requirements.txt

where the requirements.txt file was generated by pip freeze in a venv where I had installed jupyterlab, notebook and pillow[1]. This installs 90 packages from binary wheels (not 193, which was a mistake involving counting dist-info directories as packages), most of which replace packages built (needlessly) by Sage from sdists. Doing this allows me to remove the redundant secondary venv that I used for SageMath-10.2, reducing the size of the final disk image by 100MB, back to 1.0 GB for the Intel build. And it means that the jupyterlab and notebook packages are fully up-to-date [2] for the macOS binary package. The requirements.txt file will be part of the 3-manifolds/Sage_macOS repository and will serve as the means of pinning versions going forward.

Everything seems to be working, with the exception of some intermittent failures with the ipywidgets interact function / decorator. That seems to usually work reliably in jupyterlab, but often fails in notebook, but will start working in notebook if you switch to jupyterlab and back. When I did manage to make it fail in jupyterlab the jupyterlab log console (a feature which does not exist in notebook as far as I can tell) reports that a "comm" identified by some hash value does not exist. Debugging minimized obfuscated javascript code generated by the Typescript compiler is not something I am capable of, so I think I have to live with this flakiness for the moment. (I would be happy to post an issue at jupyter, but I am not able to reproduce this failure when Sage is not involved, so I don't think they would be willing to work on it.)

[1] Using a binary wheel for pillow allows me to bypass the issue of compiling openjpeg and provides all of the image formats that pillow supports.

[2] Without a change in Sage policies it is not possible to distribute an up-to-date version of jupyter-server with Sage. The reason is that the jupyter-server package depends on pyrpds, which is a python wrapper for a Rust library. Under the current policies, pyrpds would have to be built from an sdist, which would require embedding a Rust toolchain within the Sage distribution. If one is connected to the internet it is an easy one-liner to install a Rust toolchain in one's home directory, but I would have no idea how to go about embedding such a thing in Sage, and my impression is that the Sage community would not be willing to consider doing that anyway. I think that it would be wise for Sage to consider the possibility of having a new type of spkg which is installed from a binary wheel. I believe that would be a simple change, if people were willing to consider it, involving just adding a new "Sage distribution helper".

@dimpase
Copy link
Member

dimpase commented Mar 12, 2024

The "new type of package" here is just a standard pip package (not allowed under the current policy).

As you know, I have proposed to lift this restriction on sage-devel.
You are most welcome to post a follow-up supporting this proposal.

@culler
Copy link
Contributor Author

culler commented Mar 12, 2024

The "new type of package" here is just a standard pip package (not allowed under the current policy).

Sage's "standard pip packages", as currently defined, must use "none-any" wheels and must be built from an sdist unless they are pure python packages. This new type would use a platform-specific binary wheel containing shared libraries (presumably manylinux in the case of linux systems). Perhaps you were using the term "standard pip package" to mean the sort of package that one gets by just running "pip install". But Sage has its own clear definition of the term, so using the same term with a different meaning is likely to be confusing. I don't know who, if anyone, understood that you were talking about using platform-specific binary wheels in your proposal.

@culler
Copy link
Contributor Author

culler commented Mar 12, 2024

I don't think it is likely to be productive to discuss something like this in the abstract, with the whole group. To make it happen I think it would be necessary to fork Sage and implement the needed changes and demonstrate that it works. If that were done it would be a concrete proposal and it would be very clear what is being proposed. I don't think it would require any major changes, either.

@culler
Copy link
Contributor Author

culler commented Mar 12, 2024

There is one sort of major thing that would need to change: namely the documentation requirements. Matthias would like to expand the documentation to explain how each package fits into Sage. Realistically, though, with a package like pyrpds, the best we could hope to do is say "we need this because we provide jupyter notebooks and they require it".

@dimpase
Copy link
Member

dimpase commented Mar 13, 2024

The "new type of package" here is just a standard pip package (not allowed under the current policy).

Sage's "standard pip packages", as currently defined, must use "none-any" wheels and must be built from an sdist unless they are pure python packages.

This new type would use a platform-specific binary wheel containing shared libraries (presumably manylinux in the case of linux systems). Perhaps you were using the term "standard pip package" to mean the sort of package that one gets by just running "pip install".

You are right. I keep thinking of the latter, as my main examples of packages that ought to be like this are pure python packages.
E.g. pytest and tox, two optional packages which are pip packages, are perfect candidates for such packages.
The current state of affairs is that if they become standard then they must be "properly" vendored, with all its extra dependencies.
(but that's insane - where else one must do that, don't just do "pip install pytest" - unless that's something prescribed by various business/security considerations, which is really none of Sage's business, and with optional packages which can come straight from pypi it's funny to talk about "security".)

But Sage has its own clear definition of the term, so using the same term with a different meaning is likely to be confusing. I don't know who, if anyone, understood that you were talking about using platform-specific binary wheels in your proposal.

My proposal is about a possibility to have a pip standard package. TBF, I didn't think at all about this technicality, I'd be happy to just have pure Python standard pip packages.

But, indeed, rust is getting more and more popular in Python world, and more and more packages would use it, making them rather unpleasant to ship as sources (the last thing Sage needs it an extra stress of packaging a rust toolchain)

@dimpase
Copy link
Member

dimpase commented Mar 13, 2024

I don't think it is likely to be productive to discuss something like this in the abstract, with the whole group. To make it happen I think it would be necessary to fork Sage and implement the needed changes and demonstrate that it works. If that were done it would be a concrete proposal and it would be very clear what is being proposed. I don't think it would require any major changes, either.

Why? There are clearly packages which are ripe for becoming pip standard packages, e.g. pytest.

Given that Sage is the only known to me open-source Python project packaging compiler toolchains and Jupyter, I really doubt one needs a prototype. Besides, prototypes exist in few Linux distros: Gentoo, Arch, Void (and in Conda) - prototypes which only use the core sagelib, whereas the rest comes straight from the OS (or conda binary installs, in Conda case).
Your macOS app is a prototype like this too, although you use a bit more sage distro than the latter

@vbraun vbraun closed this as completed in e850ac9 Mar 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants