-
-
Notifications
You must be signed in to change notification settings - Fork 435
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
spkg-configure.m4 for most external Python pkgs #36777
Conversation
I don't want to install all that crap to test but the new |
@dimpase If you are doing this exercise as an approach for using the system Jupyter notebook / JupyterLab, I'd suggest to consider a different approach. The front-end does not need to run in the same Python instance as the kernel running the Sage library. |
Is the last comment meant to derail the effort to unvendor a bunch of packages? |
the frontend certainly does not need to run on the same Python, that is by the design of Jupyter. I don't quite know why in the 1st place it was decided to vendor the whole Jupyter in Sage, probably as a replacement of sagenb, which actually was at some point made a separate package with a separate git repo. So we are repeating the whole Jupyter packaging effort here 😭 |
SPKGs don't come with any metadata about how they're supposed to work with python, and the sage package manager doesn't do dependency resolution, so to do it right, the people interested in not using these packages would have to maintain the list by hand (like we already do with |
That's why people interested in sage-the-distro should get a separate repo where they can have their own fun with 400+ packages with built from source compilers, jupyter, whatever. I just want sagelib as a Python package, without any toolchain or frontend, scipy-style. Such a Python package can very well be installed in sage-the-distro. Or they can vendor it... |
Glad we can agree on this! The next step is to understand that this means that you can treat it like system packages providing executables instead of a Python site packages. The model of |
The problem with this is that there's no way to know what dependencies are unique to tox except to muck around in |
I have a robust solution for this in #36766 |
I'm surprised about this example. How would you describe scipy's approach to vendoring? |
scipy does not vendor compilers (or jupyter). It assumes that anyone wanting to build it from source has all the needed tools installed: python, cython, fortran, C, C++, etc... It surely provides instructions for all this. |
Is there something else about vendoring that you may be forgetting? |
Like? |
Libraries? |
sure they e.g. ship libgfortran in their binary wheels. So? |
Other libraries? |
we are talking about "toolchain", not maths stuff that needs to be delivered to the end user one way or another. yes, they vendor about a dozen things. |
i don't think we are.
They do! And in fact they even do that in the real meaning of "vendoring", namely including upstream source code in their source code distribution. So when you were saying ....
... do you mean this is what you would sagelib to look like? |
There are better examples than scipy I'm sure, but now that scipy has a meson build system the bundling will get resolved eventually: scipy/scipy#17751 For obscure math subprojects there's always a chicken and egg problem trying to convince upstream to unbundle things that no distro packages it because everyone bundles it because no distro packages it...
Aren't you still manually curating the jupyter package list? In any case, the question isn't whether or not it can be made robust, but whether or not it's a waste of time. |
Thanks for the pointer! |
anyhow, they don't vendor python, openblas, numpy. They have few header-only C++ libs as git submodules (boost/math), an LP solver Higs. They don't have 100 packages per release to update. Our technology for not vendoring is actually better, we can e.g. skip boost. |
The Sage library does not vendor anything at all. Edit: @dimpase replied by editing: I count shipping a tarfile along, and installing it as vendoring. It's merely a different wrapping of the same thing, we could have just as well checked that code into our git tree. |
So, Dima, once again: when you were saying ....
What is it that you mean? |
I use scipy as an example as I know it a bit. I got my hands dirty fixing few things for them (a side effect is that I ended up as an official contributor, in a paper, and Scopus counts about 15000 citations to this paper. Beancounters are happy :-)) |
now that's news to me ;) |
How come? What's the difference between having the package X untarred in advance, or untarred on the fly, if your building process builds/installs X? |
Matthias, years before you started actively contributing to Sage, we referred to creating spkg-configure.m4 file for an spkg X as unvendoring of X. |
Not that this matters, but no, Dima, the historical record does not agree with this: https://github.com/sagemath/sage/issues?q=unvendor |
That's perhaps a bit harsh for this discussion? |
When a project is making changes, it is important to take the developer community with you. One such route from monolithic Sage to multirepo Sage (if the community decides to do this) would go like this:
I have been working since 2020 on step 1 in #35095. This is where the real work of modularization is happening; and there's still nontrivial work to be done; and during this time, I do not have to bother other developers with the added complexity of changing source code locations. The other steps are trivial. |
This is largely orthogonal to the need to spin off Sage the distro into a number of separate repos, one or more. |
Sage the distro does not even consist of much code at all; but it provides the package metadata (versions in install-requires) that are needed for bootstrapping the pkgs/ subdirectories. |
If you count the code which ends up installed by the spkg-install and similar scripts, it is a lot of code (e.g. gcc/gfortran are a lot of code) |
Would build/pkgs still bother you if everything except for the patches was just one big yaml file? |
do you want to replace the tarballs in |
Update: they are not going to be in 2.2.0a2 - aleaxit/gmpy#462 (comment) but there is an interesting development regarding an openblas wheel - aleaxit/gmpy#462 (comment) |
incidentally we can also reuse the related
to get rid of gcc/gfortran |
…sions of the Sage distribution <!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> Discussions of the complexity of the Sage distribution pop up unexpectedly, as seen in sagemath#36982 (comment), sagemath#36982 (comment), sagemath#36982 (comment), sagemath#36982 (comment), sagemath#36982 (comment), sagemath#36982 (comment), sagemath#36982 (comment) sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment), sagemath#36726 (comment) sagemath#36777 (comment), sagemath#36777 (comment), sagemath#36777 (comment), sagemath#36777 (comment), sagemath#36777 (comment), sagemath#36777 (comment) To help put such discussions on a solid factual basis, we introduce the command `sage --package metrics`. ``` $ ./sage -package metrics :standard: has_file_distros_arch_txt=131 has_file_distros_conda_txt=216 has_file_distros_debian_txt=125 has_file_distros_fedora_txt=138 has_file_distros_gentoo_txt=181 has_file_distros_homebrew_txt=61 has_file_distros_macports_txt=129 has_file_distros_nix_txt=51 has_file_distros_opensuse_txt=146 has_file_distros_slackware_txt=25 has_file_distros_void_txt=184 has_file_patches=35 has_file_spkg_check=59 has_file_spkg_configure_m4=222 has_file_spkg_install=198 has_tarball_upstream_url=231 line_count_file_patches=22561 line_count_file_spkg_check=402 line_count_file_spkg_configure_m4=2792 line_count_file_spkg_install=2960 packages=272 type_standard=272 ``` Use `PATH=build/bin:$PATH SAGE_ROOT=some-other-worktree build/bin/sage- package metrics :standard:` to obtain the metrics of another version of Sage in some other worktree. We add computation and before/after comparison of the metrics to the CI Linux Incremental workflow. As an illustration, we change one Python package from "normal" to "wheel", removing an `spkg-install.in` file in the process. See https:// github.com/sagemath/sage/actions/runs/7342841283/job/19992606617?pr=3697 7#step:6:12 More metrics can be added after - sagemath#36740 <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#36977 Reported by: Matthias Köppe Reviewer(s): Kwankyu Lee, Matthias Köppe
Documentation preview for this PR (built with commit 016fc0e; changes) is ready! 🎉 |
After #36129 and #36776 we are still left with a bunch of external (non-Sage) Python packages
which need
spkg-configure.m4
and distros info.Here we add these.
📝 Checklist
⌛ Dependencies