Meta-ticket: Add external user packages as optional/experimental packages

[mkoeppe]

Since Sage 9.2, the spkg section of the Sage reference manual is automatically generated from the information in the SAGE_ROOT/build/pkgs/ directory. Each SPKG is represented by a subdirectory. The structure of these subdirectories is explained in the Developer's Guide.

This meta-ticket tracks the task of adding external packages, in particular those that used to be listed at https://wiki.sagemath.org/SageMathExternalPackages, as optional/experimental packages.
By including them as optional/experimental packages in build/pkgs:

• They are automatically included in our reference manual (see Improve build/pkgs/*/SPKG.rst #29655)
• Information about them can be obtained using the commands
  • Optional packages available: ./sage -optional
  • Experimental packages available: ./sage -experimental
  • Details about an SPKG: ./sage -info <SPKG>
• The GH Actions workflows CI Linux and CI macOS will run them on every Sage development release, providing continuous integration that will allow use to catch unintended breaking changes during the Sage development cycle

Already added as packages, see spkg section of the Sage reference manual:

• Add slabbe as an optional package #31237 slabbe: Sébastien Labbé's research code (blog post, 2014 blog post, GitLab.com repo)
• Add admcycles as an optional package #31238 admcycles: Computation in the tautological ring of the moduli space of curves by Johannes Schmitt et al. (GitLab.com repo)
• Add ore_algebra optional package #31239 ore_algebra: Ore algebras and Ore polynomials by Manuel Kauers et al. (GitHub repo)
• Add surface-dynamics as a pip package #31224 surface_dynamics: Dynamics on surfaces (measured foliations, interval exchange transformation, Teichmüller flow, etc.) by Vincent Delecroix and Samuel Lelièvre (GitHub repo)
• sage-flatsurf pip package #31240 sage_flatsurf: Computation with flat surfaces by Vincent Delecroix and Pat Hooper (GitHub repo)
• Add package phitigra: Graph editor that works with Jupyter #30540 phitigra: Graph editor for SageMath as a Jupyter widget by Jean-Florent Raymond (GitHub repo)

Tickets for pip-installable packages:

The packages will typically be added as pip packages instead of normal packages.
These do not come with tarball information and do not have to pin the version, so by default the latest version on PyPI would be installed.
Hence there is no additional maintenance burden from updating the packages.
Adding the packages is made easy by the new sage -package create commands, added in Sage 9.3.

• Implement Riemann theta functions #6371 RiemannTheta: Riemann Theta function numerical evaluation by Nils Bruin, Sohrab Ganjian. Fairly small straightforward package with some cython code. Should probably be integrated into SageMath proper at some point to guarantee it stays up-to-date and functional (Github repo)

• Add sage-numerical-interactive-mip, cutgeneratingfunctionology as pip packages #31235 sage-numerical-interactive-mip: Interactive mixed integer linear programming solver by Peijun Xiao, Zeyi Wang, and Yuan Zhou (GitHub repo)

• Add sage-numerical-interactive-mip, cutgeneratingfunctionology as pip packages #31235 cutgeneratingfunctionology (formerly infinite-group-relaxation-sage-code): Computation and experimentation with cut-generating functions, in particular the Gomory-Johnson model by Matthias Köppe, Yuan Zhou, C. Y. Hong, and Jiawei Wang (GitHub repo)

• Usainboltz package: fast Boltzmann random generation of tree-like structures #27526 usainboltz: Random generation of tree-like structures by Matthieu Dien and Martin Pépin (GitLab.com repo)

• Add texsurgery as a pip package, advertise as an alternative to sagetex #33778 texsurgery

• Meta-ticket: Add packages from 3-manifolds / computop #31176 Add packages from 3-manifolds / computop (Add snappy as a pip package #31180: snappy)

• Add sage-train-track as an optional package #31223 train_track: Free group automorphisms and train-track representatives by Thierry Coulbois (GitHub repo)

• Add multipolynomial-bases as an optional package #31416 multipolynomial-bases: Multipolynomial bases (Schubert, Grothendieck, Key) by Viviane Pons (GitHub repo)

No ticket yet:

• sage_acsv: Algorithms for analytic combinatorics in several variables by Benjamin Hackl et al.

• oriented_matroids by Aram Dermenjian

• flagmatic: Tool for researchers in extremal graph theory by Emil R. Vaughan et al. (orig GitHub repo, GitHub repo)

• stallings_graphs: experiment with finitely generated subgroups of infinite groups by Pascal Weil (self-hosted GitLab repo)

• addcombq: Fast Additive Combinatorics functions by Matt Torrence (GitHub repo)

• arithmat: Implementation of arithmetic matroids and toric arrangements by Roberto Pagaria and Giovanni Paolini (GitHub repo)

• gcaops: Kontsevich Graph Complex Action on Poisson Structures by Ricardo Buring (GitHub repo)

• cvolume: Compute completed volumes of strata of quadratic differentials with odd zeros by Eduard Duryev
  (GitHub repo)

• badic: Beta-adic tools and automata by Paul Mercat (GitLab.com repo)

• abelfunctions: Computing with Abelian functions, Riemann surfaces, and algebraic curves by Chris Swierczewski (GitHub repo, discussion)

• carlin: Carleman linearization of polynomial differential equations by Marcelo Forets (GitHub repo)

• adeles: Computing with adèles and idèles by Mathé Hertogh (GitHub repo)

• yacop_sage: Steenrod algebra cohomology by Christian Nassau (GitHub repo)

• FlexRiLog: Flexible and rigid labelings of graphs by Jan Legerský (GitHub repo)

• sage_cluster_pictures: Cluster pictures, BY-trees, models and local invariants of hyperelliptic curves by Alex J. Best and Raymond van Bommel (GitHub repo)

• dual-pairs: Computing with dual pairs of algebras by Peter Bruin (GitLab.com repo)

• msinvar: Computation of moduli space invariants by Sergey Mozgovoy (GitHub repo)

• comb_walks: Walk Models with elliptic kernel function by Frédéric Chyzak, Antonio Jiménez-Pastor, Pierre Lairez (self-hosted GitLab repo)

• kerrgeodesic_gw: Computing geodesics of the Kerr spacetime as well as gravitational waves from bodies orbiting a Kerr black hole by Eric Gourgoulhon, Alexandre Le Tiec, Frédéric H. Vincent and Niels Warburton (GitHub repo)

• dd_functions: Computation with DD-finite functions by Antonio Jiménez-Pastor (GitHub repo)

• sherali_adams: Sherali-Adams relaxation on a numpy system of the form Ax <= b by Matthew Drescher (GitHub repo)

• conrey-dirichlet-characters: Working with Dirichlet characters using a numbering scheme described by Brian Conrey by Jonathan Bober (GitHub repo)

• pychebfun: Python Chebyshev Functions by Olivier Verdier, Chris Swierczewski (GitHub repo, orig GitHub repo)

• pysemigroup: Automata and semigroups by Charles Paperman (self-hosted GitLab repo)

• zetalib: Computing zeta functions of groups, algebras, and modules by Tobias Rossmann, Tomer Bauer (GitLab.com repo; orig website (404))

• hypigu: Compute the Igusa local zeta function and flag Hilbert–Poincaré series of hyperplane arrangements by Joshua Maglione (GitHub repo)

• mclf: Computing with Models of Curves over Local Fields by Stefan Wewers, Julian Rüth (GitHub repo)

• hilbert-modular-group: Computing with Hilbert modular groups, including a reduction algorithm by Fredrik Stromberg (GitHub repo)

Not pip-installable:

In this ticket, only pip-installable packages will be added.
This adds an incentive to package authors to bring their packages to this form.

• starproducts: Symbolic calculations with Poisson brackets and Kontsevich's deformation quantization formula by Peter Banks and Brent Pym (Bitbucket repo)
• hodge-diamond-cutter: Hodge diamonds (and Hochschild homology) of smooth projective varieties by Pieter Belmans (GitHub repo)
• Torus Knots Counter: computes the motive of the irreducible SL(r,k)-character variety of an (n,m)-torus knot by Ángel González-Prieto and Vicente Muñoz (GitHub repo with Jupyter notebook)
• ResSing: Calculating regular models of curves #28651: ResSing: Calculating regular models of curves by Suchandan Pal (zip file with .sage files)
• (Local) Igusa and topological zeta functions for Newton-non-degenerated polynomials by Juan Viu-Sos (GitHub repo)
• ecfactory: Construct elliptic curves with certain properties by http://www.scipr-lab.org/ (GitHub repo)

Maintained/abandoned on a Trac ticket:

• Chow #27228: Chow: computations in intersection theory by Christoph Sorger and Manfred Lehn

Outdated (disappeared or does not work with current Sage):

• sage-semigroups: A semigroup (representation) theory library by Nicolas M. Thiéry (GitHub repo, broken as of 2022-09, Sage 9.7)
• CHA: Combinatorial Hopf algebras by Nicolas Borie (Bitbucket repo - 404)
• cryptosage: Cryptography algorithms by Sara Forouhar (GitHub repo with .sage files, last updated 2015)
• sage-drg: Checking the feasibility of distance-regular graph parameter sets by Janos Vidali (GitHub repo, not pip-installable, no Python 3 support)
• SchottkyUniformization: SK prime function, Riemann Theta function built from a RS via Schottky uniformization by Jeremy Upsal (GitHub repo, last updated 2016)
• ss-isogeny-software: Quantum-Resistant Cryptosystems from Supersingular Elliptic Curve Isogenies by Luca De Feo (GitHub repo, marked as obsolete by the author)
• lecf: Lyapunov exponents for multidimensional continued fractions by Vincent Delecroix and Sébastien Labbé (GitHub repo - last updated 2014, GitHub repo - 404)
• harmonic-modules: Computing the bicharacter of diagonal harmonic polynomials by Nicolas Thiéry (GitHub repo)

Discussions:

• https://groups.google.com/g/sage-devel/c/ZNyLBKmc0EE/m/oUll3liBAwAJ - on version metadata

---

-------------------------
List of external packages 
-------------------------

This list used to be maintained at https://wiki.sagemath.org/SageMathExternalPackages

'''Please help by moving packages to one of the sections above!'''

See also https://github.com/slel/external-packages
for another collection

`Modular Abelian Varieties <https://github.com/williamstein/sage_modabvar>`_
-----------------------------------------------------------------------------

  by William Stein and Hao Chen

  Links:

  - https://github.com/williamstein/sage_modabvar
  - https://github.com/haochenuw/sage_modabvar


`Purple Sage <https://github.com/fredstro/psage>`_
------------------------------------------------------

  by William Stein, Fredrik Stromberg, Stephan Ehlen, Martin Raum and others

  A Library of Number Theory Code that depends on Sage.



`fgb_sage <https://github.com/mwageringel/fgb_sage>`_
-----------------------------------------------------

  by Markus Wageringel

  Sage interface for computing Gröbner bases with FGb.

`guptri_py <https://github.com/mwageringel/guptri_py>`_
-------------------------------------------------------

  by Markus Wageringel

  Sage/NumPy interface for solving generalized eigenvalue problems of singular matrix pencils with GUPTRI.


Brocoli
-------

  by Jean-Philippe Labbé

  Package dealing with LImit ROots of COxeter groups (eternal Beta version)

  - home: http://page.mi.fu-berlin.de/labbe/pages/brocoli.html
  - docs-html: https://jplab.github.io/brocoli/doc/html
  - docs-pdf: http://page.mi.fu-berlin.de/labbe/pdfs/brocoli.pdf
  - gallery: http://page.mi.fu-berlin.de/labbe/gallery.html
  - repo: https://github.com/jplab/brocoli

`sagemath-polytope-tikz <http://page.mi.fu-berlin.de/labbe/polytope.html>`_
---------------------------------------------------------------------------

  by Jean-Philippe Labbé

`SFBpresentation <https://github.com/jplab/SFBpresentation>`_
-------------------------------------------------------------

  by Jean-Philippe Labbé
  
  

--------
See also
--------

- `Packages on PyPI matching 'sagemath'
  <https://pypi.org/search/?q=sagemath>`_
- `Packages on PyPI matching 'sage' and Mathematics <https://pypi.org/search/?q=sage&o=&c=Topic+%3A%3A+Scientific%2FEngineering+%3A%3A+Mathematics>`_
- `Packages on PyPI matching 'sage' and Science <https://pypi.org/search/?q=sage&o=&c=Intended+Audience+%3A%3A+Science%2FResearch>`_

- `Packages on github matching "Sage code" <https://github.com/search?q=%22Sage+code%22>`_
- `Packages on gitlab matching "sagemath" <https://gitlab.com/search?search=sagemath>`_
- `Packages on Google Code Archive with tag "SAGE" <https://code.google.com/archive/search?q=domain:code.google.com%20label:SAGE>`_
- `Packages on Bitbucket matching "sagemath" <https://bitbucket.org/repo/all?name=sagemath>`_

CC: @slel @mwageringel @videlec @seblabbe @sagetrac-tmonteil @jplab @fredstro @fchapoton @sagetrac-srobik @williamstein @nthiery @saraedum @edgarcosta @sorger-c @mforets @pjbruin @rburing @mathzeta

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/31164

[videlec]

comment:1

Before proceeding, I would like to understand what package developers and package users would gain by declaring Python packages as sage optional packages. For now, I would only consider doing this if

• package versions were not tight to sage versions. In the curent setup a package upgrade needs a ticket review and, when merged, has to wait for a new sage version.
• there is no serious continuous integration for packages (as they have eg for gap)

[mkoeppe]

comment:2

Replying to @videlec:

Before proceeding, I would like to understand what package developers and package users would gain by declaring Python packages as sage optional packages. For now, I would only consider doing this if

• package versions were not tight to sage versions. In the curent setup a package upgrade needs a ticket review and, when merged, has to wait for a new sage version.

The solution here is to use pip packages instead of normal packages.
https://doc.sagemath.org/html/en/developer/packaging.html#package-source-types

These do not come with tarball information and do not have to pin the version, so by default the latest version on PyPI would be installed.

• there is no serious continuous integration for packages (as they have eg for gap)

Well, we do have workflows on GH Actions that at least tries to install all optional and experimental packages!

• https://github.com/sagemath/sage/actions/runs/447854806
• https://github.com/sagemath/sage/actions/runs/447854801

[mkoeppe]

Reviewer: https://github.com/mkoeppe/sage/actions/runs/489285128

[mkoeppe]

comment:20

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.

[dbenielli]

Hi Sage Team,

Olivier Ramare (https://ramare-olivier.github.io/) from I2M (Institut de Mathematique de Marseille and I (working for Institut Archimède https://labex-archimede.univ-amu.fr/) worked to make the work on the Eulerian products developed by Olivier and all accessible via an sage math external package. By the way a fast multi-precision computation of some Euler products is provided.
The documentation is accessible on https://archimede-institut.github.io/sage-euler-product/, and the code are visible on github https://github.com/archimede-institut/sage-euler-product, The package is not yet declared in pypi, but that should be done quickly.

Can you tell me the procedure for declaring this external package as experimental in sagemath?

Thanks

Dominique

[mkoeppe]

@dbenielli Apologies for missing this message earlier. Once you have a release on PyPI, please open a PR for Sage that adds this package as a "pip" package. See https://doc.sagemath.org/html/en/developer/packaging.html#creating-packages