Restructure sage.*.all for modularization, replace relative by absolute imports

[mkoeppe]

We restructure the all.py files for modularization.

Guided by the technical constraints outlined in https://groups.google.com/g/sage-devel/c/ozh-7ZZ848s, #35095 defines distribution packages (pip-installable packages) sagemath-{brial, combinat, eclib, flint, gap, giac, glpk, graphs, groups, homfly, lcalc, libbraiding, libecm, linbox, modules, mpmath, ntl, pari, plot, polyhedra, schemes, singular, standard-no-symbolics, symbolics}.

When a namespace package such as sage.misc is filled by modules from several distribution packages, we create modules named:

• src/sage/misc/all__sagemath_environment.py
• src/sage/misc/all__sagemath_objects.py
• src/sage/misc/all__sagemath_repl.py

Import statements are moved from src/sage/misc/all.py to these files as appropriate, and src/sage/misc/all.py imports * from there.

Also some imports are replaced by lazy imports.

The new files provide the top level namespaces for the modularized distribution packages, thus enabling modularized testing.

This design was introduced in #29865 (merged in Jan 2022, early in the Sage 9.6 development cycle).

• Copied from Modularization of sagelib: Distributions sagemath-{brial,combinat,eclib,flint,gap,giac,glpk,graphs,groups,homfly,lcalc,libbraiding,libecm,linbox,modules,mpmath,ntl,pari,plot,polyhedra,schemes,singular,standard-no-symbolics,symbolics} #35095.
• Part of Meta-ticket: Modularize sagelib into separate distributions (pip-installable packages) sagemath-... for Sage 10.x #29705

Moreover, applied a one-line command to replace relative by absolute imports, thus complementing #36666, which does not touch .all* files.

The changes to other files in sage.modules etc. come from the PRs #36597, #36572, #36588, #36589 merged here and do not need review.

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

• Depends on Replace relative imports by absolute ones in modules #36597 (merged here to resolve merge conflict)
• Depends on Replace relative imports by absolute ones in categories #36572 (merged here to resolve merge conflict)
• Depends on Replace relative imports by absolute ones in rings #36588 (merged here to resolve merge conflict)
• Depends on Replace relative imports by absolute ones in a few packages #36589 (merged here to resolve merge conflict)
• Depends on Autogenerate flint headers #36449 (merged here to resolve merge conflict)
• Depends on add simple function to count irreducible polynomials over 𝔽q #36390 (merged here to resolve merge conflict)

[dcoudert]

The coding style can certainly be improved and the syntax used when importing multiple methods from a module could be unified.

I tried on Fedora 35 and it seems ok.

[mkoeppe]

I would be in favor of running tools like autopep8, ruff, or black on these files to implement a consistent style. I don't have a personal preference for any particular style, nor do I regularly use these tools for auto-formatting. So discussion, suggestion, help are all welcome.

[tobiasdiez]

I don't really like all these all files. They add a lot of noise without providing much added value. I thought a bit about alternatives and came up with the following idea:

• Each distribution sets a global constant (say IS_CATEGORIES_DISTRO for the categories distro)
• In the single all file, you then have blocks that conditionally import certain modules depending on these global flag.

In this way you can introduce as many distributions as you want without having many all files. This is similar to the TYPE_CHECKING flag used by pyright.

What do you think?

[tobiasdiez]

Why convert this to a lazy import?

[mkoeppe]

It was convenient in the modularization. Do you have a specific concern about it?

[jhpalmieri]

For another ticket: I think that SteenrodAlgebra (which I wrote) and other named algebras perhaps could be deprecated and accessed solely through the algebras catalog, as a way of cleaning up the namespace. (I have no concerns about lazy importing here, for what that's worth.)

[tobiasdiez]

It was convenient in the modularization. Do you have a specific concern about it?

Just that it makes it very hard to review this PR, as there are all these unrelated changes sprinkled in. Can you extract these lazy_import changes to a new PR please?

[mkoeppe]

No, I don't believe this would be helpful for anything.

[tobiasdiez]

Can you please revert these changes to modules that you don't touch otherwise. Just increases the diff unnecessarily and introduces merge conflicts with other PRs. E.g. these changes here are already included in #36572.

[mkoeppe]

The point of the PR is to replace all the all files with the version provided here.

[mkoeppe]

introduces merge conflicts with other PRs. E.g. these changes here are already included in #36572.

When the same change appears, git automatically resolves it.

[tobiasdiez]

The point of the PR is to replace all the all files with the version provided here.

How is this change relevant to the modularization?

[mkoeppe]

This change is part of "replace relative by absolute imports", which is in the title of the PR.

[mkoeppe]

Update: The PR mentioned above, #36572, was merged in develop.

[tobiasdiez]

There is still no logical connection between making the imports absolute and the modularization. Both touch the all files yes, but that doesn't mean they should be in the same PR.

[mkoeppe]

I thought a bit about alternatives and came up with the following idea:

• Each distribution sets a global constant (say IS_CATEGORIES_DISTRO for the categories distro)
• In the single all file, you then have blocks that conditionally import certain modules depending on these global flag.

In this way you can introduce as many distributions as you want without having many all files. This is similar to the TYPE_CHECKING flag used by pyright.

What do you think?

This idea does not work because after an all module is imported, it is in sys.modules and a subsequent import of it is a no-op.

[mkoeppe]

I would be in favor of running tools like autopep8, ruff, or black on these files to implement a consistent style. I don't have a personal preference for any particular style, nor do I regularly use these tools for auto-formatting. So discussion, suggestion, help are all welcome.

In particular: https://stackoverflow.com/questions/76771858/ruff-does-not-autofix-line-too-long-violation

So there is a choice between the style that autopep8 uses (autopep8 --in-place --select E501 --max-line-length=88 src/sage/**/all*.py):

diff --git a/src/sage/schemes/hyperelliptic_curves/all.py b/src/sage/schemes/hyperelliptic_curves/all.py
index e314dceb1e0..40c8b58e823 100644
--- a/src/sage/schemes/hyperelliptic_curves/all.py
+++ b/src/sage/schemes/hyperelliptic_curves/all.py
@@ -41,6 +41,7 @@ lazy_import('sage.schemes.hyperelliptic_curves.invariants',
             ['igusa_clebsch_invariants', 'absolute_igusa_invariants_kohel',
              'absolute_igusa_invariants_wamelen', 'clebsch_invariants'],
             deprecation=28064)
-from sage.schemes.hyperelliptic_curves.mestre import (Mestre_conic, HyperellipticCurve_from_invariants)
+from sage.schemes.hyperelliptic_curves.mestre import (
+    Mestre_conic, HyperellipticCurve_from_invariants)
 from sage.schemes.hyperelliptic_curves import monsky_washnitzer
 del lazy_import
diff --git a/src/sage/stats/hmm/all.py b/src/sage/stats/hmm/all.py
index d331f45b41c..2d88d943fb1 100644
--- a/src/sage/stats/hmm/all.py
+++ b/src/sage/stats/hmm/all.py
@@ -8,6 +8,7 @@
 # We lazy_import the following modules since they import numpy which slows down sage startup
 from sage.misc.lazy_import import lazy_import
 lazy_import("sage.stats.hmm.hmm", ["DiscreteHiddenMarkovModel"])
-lazy_import("sage.stats.hmm.chmm", ["GaussianHiddenMarkovModel","GaussianMixtureHiddenMarkovModel"])
+lazy_import("sage.stats.hmm.chmm", [
+            "GaussianHiddenMarkovModel","GaussianMixtureHiddenMarkovModel"])
 lazy_import("sage.stats.hmm.distributions", ["GaussianMixtureDistribution"])
 del lazy_import

and what black uses (black src/sage/**/all*.py):

--- a/src/sage/schemes/hyperelliptic_curves/all.py
+++ b/src/sage/schemes/hyperelliptic_curves/all.py
@@ -37,10 +37,21 @@ from sage.misc.lazy_import import lazy_import
 
 from sage.schemes.hyperelliptic_curves.constructor import HyperellipticCurve
 from sage.schemes.hyperelliptic_curves.kummer_surface import KummerSurface
-lazy_import('sage.schemes.hyperelliptic_curves.invariants',
-            ['igusa_clebsch_invariants', 'absolute_igusa_invariants_kohel',
-             'absolute_igusa_invariants_wamelen', 'clebsch_invariants'],
-            deprecation=28064)
-from sage.schemes.hyperelliptic_curves.mestre import (Mestre_conic, HyperellipticCurve_from_invariants)
+
+lazy_import(
+    "sage.schemes.hyperelliptic_curves.invariants",
+    [
+        "igusa_clebsch_invariants",
+        "absolute_igusa_invariants_kohel",
+        "absolute_igusa_invariants_wamelen",
+        "clebsch_invariants",
+    ],
+    deprecation=28064,
+)
+from sage.schemes.hyperelliptic_curves.mestre import (
+    Mestre_conic,
+    HyperellipticCurve_from_invariants,
+)
 from sage.schemes.hyperelliptic_curves import monsky_washnitzer
+
 del lazy_import
diff --git a/src/sage/stats/hmm/all.py b/src/sage/stats/hmm/all.py
index d331f45b41c..6c879e12f5d 100644
--- a/src/sage/stats/hmm/all.py
+++ b/src/sage/stats/hmm/all.py
@@ -7,7 +7,11 @@
 
 # We lazy_import the following modules since they import numpy which slows down sage startup
 from sage.misc.lazy_import import lazy_import
+
 lazy_import("sage.stats.hmm.hmm", ["DiscreteHiddenMarkovModel"])
-lazy_import("sage.stats.hmm.chmm", ["GaussianHiddenMarkovModel","GaussianMixtureHiddenMarkovModel"])
+lazy_import(
+    "sage.stats.hmm.chmm",
+    ["GaussianHiddenMarkovModel", "GaussianMixtureHiddenMarkovModel"],
+)
 lazy_import("sage.stats.hmm.distributions", ["GaussianMixtureDistribution"])
 del lazy_import

no work items

[tobiasdiez]

I thought a bit about alternatives and came up with the following idea:

• Each distribution sets a global constant (say IS_CATEGORIES_DISTRO for the categories distro)
• In the single all file, you then have blocks that conditionally import certain modules depending on these global flag.

In this way you can introduce as many distributions as you want without having many all files. This is similar to the TYPE_CHECKING flag used by pyright.
What do you think?

This idea does not work because after an all module is imported, it is in sys.modules and a subsequent import of it is a no-op.

Can you explain why one would import the all files multiple times? Is it not enough to import it once, determine which distros are installed and then import the correct submodules based on this info?

[mkoeppe]

Is it not enough to import it once, determine which distros are installed and then import the correct submodules based on this info?

No, the main purpose of the "all" files is to fill the top-level namespace. But the contents of a module such as sage.all should really not depend on what distributions happen to be installed; that would be a very problematic design.

Instead I define separate top-levels that are suitable for the separate testing of the distributions. (I've added a link on that to the ticket description.)

Edit: As I wrote in #36524 (comment), this is part of the solution of how to test things when not all of sage.all can be imported.

[dcoudert]

The documentation don't build due to

  [sagemath_doc_html-none] OSError: /sage/src/doc/en/reference/calculus/index.rst:26: WARNING: unknown document: 'sage/calculus/expr'

[tobiasdiez]

But the contents of a module such as sage.all should really not depend on what distributions happen to be installed; that would be a very problematic design.

Why do you think this would be a problematic design? With your version you also have to determine which distros are installed and then import the corresponding all file.
Also what I'm proposing is very similar to the current modularization of the doctests using needs. Your approach here with special all files would correspond to extracting all doctests that need certain distro x to a new file with suffix _x.

[mkoeppe]

With your version you also have to determine which distros are installed and then import the corresponding all file.

No, that's not what is done. Read the documentation link I added to the ticket description.

[mkoeppe]

So there is a choice between the style that autopep8 uses [...] and what black uses [...]

I went with autopep8.

[mkoeppe]

The documentation don't build due to

  [sagemath_doc_html-none] OSError: /sage/src/doc/en/reference/calculus/index.rst:26: WARNING: unknown document: 'sage/calculus/expr'

I think I've fixed it now in 94cca3f

[kcrisman]

Matthias has accused me of being unable to read at least four times

@orlitzky That's highly inappropriate. I certainly have not.

Given comments like this:

At least read the PR description please

perhaps it's not accusing of being unable to read, but of being unwilling to do so, which in terms of this context might amount to much the same thing. It would have been more helpful to ask the necessary clarifying questions instead of reiterating "read the description", that would likely have gotten us more quickly to @orlitzky real question, which was why we have "all.py" files in the first place. (To which @jhpalmieri response is entirely appropriate.)

[mantepse]

I am now casting a vote of -1 on this PR.

The reason is that I do not properly understan the implications of the annotations of the form # sage_setup: distribution = sagemath-graphs in the various modules, and I think that many of them are plain wrong, at least from my personal mathematical perspective.

I think that separating math into classes like graph-theory, combinatorics, symbolics in a computer algebra system looks very strange and is bound to lead to problems in the long run, but I may be convinced otherwise.

[kcrisman]

I think that separating math into classes like graph-theory, combinatorics, symbolics in a computer algebra system looks very strange and is bound to lead to problems in the long run, but I may be convinced otherwise.

My understanding is that it's more about what other software is needed for the different modules (e.g., some users might not want to install Lisp to do symbolic integration via Maxima). So while there is a correlation loosely to mathematical specialties, I guess it doesn't have to be strict.

…

Message ID: ***@***.***>

[mantepse]

I think that separating math into classes like graph-theory, combinatorics, symbolics in a computer algebra system looks very strange and is bound to lead to problems in the long run, but I may be convinced otherwise.
My understanding is that it's more about what other software is needed for the different modules (e.g., some users might not want to install Lisp to do symbolic integration via Maxima). So while there is a correlation loosely to mathematical specialties, I guess it doesn't have to be strict.

I'm afraid I now have more questions than before, please bear with me :-)

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason?

My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies? (Of course, at some point dependencies might change, for example, there might be a switch from glpk to scip.)

Before I read - by chance - distribution = sagemath-graphs somewhere, I thought one would "modularize" things like the repl, user interfaces, and perhaps some low level stuff. But it seems to me now that this is really about the modularization of the mathematics.

Also, I find it hard to believe that it is about dependencies, because the stuff in abstract_tree.py and friends has no dependencies on external software (unless you want to LaTeX them, perhaps).

[kcrisman]

I think that separating math into classes like graph-theory, combinatorics,

symbolics in a computer algebra system looks very strange and is bound to lead to problems in the long run, but I may be convinced otherwise. My understanding is that it's more about what other software is needed for the different modules (e.g., some users might not want to install Lisp to do symbolic integration via Maxima). So while there is a correlation loosely to mathematical specialties, I guess it doesn't have to be strict. I'm afraid I now have more questions than before, please bear with me :-)

Sorry, I've already exhausted my knowledge :-) but hopefully Matthias or others can clarify your good questions.

…

Message ID: ***@***.***>

[mkoeppe]

If this is about dependencies on other software, why aren't the distributions named after these dependencies?

Some are. See the required reading. Links in #29705.

[mkoeppe]

My fear would be that at some point there is a request not to use symbolics in some module

@mantepse Don't fear. This has been thought through carefully, and you can read about it in the Sage developer's guide.

[culler]

I think Sage should move forward with the modularization project. I vote +1 on this PR.

[GMS103]

+1
Guillermo

[mantepse]

I posed more precise versions of the questions above in #36964 (comment). Since they have not been answered yet, and that pull request is already closed, I'll repeat them here.

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason? @kwankyu's comment above suggests that having something in the "wrong" distribution wouldn't be a big deal. But this begs the question: who profits from cutting the math library into pieces (which look very arbitrary to me and have a curious emphasis on discrete math topics)?

My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

(I don't think this fear is unjustified: in the section of the developer guide you pointed to, I find

The imports of the symbolic functions ceil() and floor() can likely be replaced by the artithmetic functions integer_floor() and integer_ceil().

OK, so some user of that module happily replaces the two functions. Now, I come along and would like to replace some other implementation by a call to something defined in symbolics. But that would be breaking a promise to the user, so it would be really hard to justify.

In fact, this happened to me already, in some sense. I noticed a function definition in sage.modular.multiple_zeta with misleading documentation, which could be replaced by a call to code in sage.combinat. However, this is already hard to do, because it might affect performance (which is a very valid point in my opinion). I think it would be extremely bad to make it even harder.

2.) If this is about dependencies on other software, why aren't the distributions named after these dependencies? (Of course, at some point dependencies might change, for example, there might be a switch from glpk to scip.)

Before I read - by chance - distribution = sagemath-graphs somewhere, I thought one would "modularize" things like the repl, user interfaces, and perhaps some low level stuff. But it seems to me now that this is really about the modularization of the mathematics.

Also, I find it hard to believe that it is about dependencies, because the stuff in abstract_tree.py and friends has no dependencies on external software (unless you want to LaTeX them, perhaps).

[NathanDunfield]

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason?

The SnapPy project is one example. This is a stand-alone Python package which gains extra functionality when used within Sage. I would much prefer we just required the (relevant parts of) the Sage math library, but we want SnapPy to be part of the broader Python ecosystem, being available on PyPI and installable via pip install snappy.

More broadly, I strongly agree with William Stein that Sage needs to be part of the broader Python ecosystem. To me, that's the main purpose of the modularization project, to fit it in better with the rest of the Scientific Python community. The correct level of granularity to choose for modularization is not an easy technical question, but I believe some level of modularization (e.g. making Lisp optional) is necessary to achieve this goal.

[mantepse]

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason?

The SnapPy project is one example. This is a stand-alone Python package which gains extra functionality when used within Sage. I would much prefer we just required the (relevant parts of) the Sage math library, but we want SnapPy to be part of the broader Python ecosystem, being available on PyPI and installable via pip install snappy.

More broadly, I strongly agree with William Stein that Sage needs to be part of the broader Python ecosystem. To me, that's the main purpose of the modularization project, to fit it in better with the rest of the Scientific Python community. The correct level of granularity to choose for modularization is not an easy technical question, but I believe some level of modularization (e.g. making Lisp optional) is necessary to achieve this goal.

Yes, I agree completely. However, this means we want to distinguish between non-python dependencies, and not between subfields of mathematics, and the current pull request does not do this, as far as I can tell.

I think I would be quite OK with having, for example, sage-without-lisp, if there is a good reason for it (even though I love lisp personally, and I think that sage without will be fairly crippled).

[mkoeppe]

Thanks @culler and @GMS103 for the support.

This brings the tally to:

• 8 in favor: @mkoeppe @kwankyu @culler @jhpalmieri @kiwifb @NathanDunfield @GMS103 @roed314
• 5 against: @tobiasdiez @dimpase @tornaria @mantepse @orlitzky

[kwankyu]

My fear would be that at some point there is a request not to use symbolics in some module, because Lisp is hard to install on some system.

That should not happen. Modularization is downstream to the sage library. Yes, we are restructuring some parts of the sage library to fit with modularization. But modularization should never be an obstacle in developing the sage library. If it ever be, the sage community might drop the modularization project.

[mkoeppe]

I think I would be quite OK with having, for example, sage-without-lisp

That one, more or less, is the distribution package sagemath-standard-no-symbolics.

[mkoeppe]

1.) Is there an example for someone who did not want to use sage because of some dependency of the math library? Or at least a possible reason?

The SnapPy project is one example. This is a stand-alone Python package which gains extra functionality when used within Sage. I would much prefer we just required the (relevant parts of) the Sage math library, but we want SnapPy to be part of the broader Python ecosystem, being available on PyPI and installable via pip install snappy.

@NathanDunfield Thanks for sharing this clear example. Would you be able to estimate how much work (if any) has been necessary to make Sage an optional dependency only? (In our June 2023 discussion in https://groups.google.com/g/sage-devel/c/lb1Eq8o0a5I/m/JJQvmPzfAQAJ, you indicated "Right now, in the stand-alone apps we backstop some of Sage's functionality (e.g. arbitrary precision floats) with CyPari".)

[saraedum]

This is now duplicated at #37900 to revert the revert #37796. Probably one of the two PRs should be closed @mkoeppe.

[mkoeppe]

Closing as this PR has been replaced by / resurrected as #37900.