From 2494538ee6cb688c65b63550921fa4321132cdf3 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 18 Nov 2021 11:32:34 -0800 Subject: [PATCH 01/30] src/doc/en/developer/packaging.rst: Update section 'Python-based packages' --- src/doc/en/developer/packaging.rst | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/src/doc/en/developer/packaging.rst b/src/doc/en/developer/packaging.rst index e91e66ed305..aaa294986d1 100644 --- a/src/doc/en/developer/packaging.rst +++ b/src/doc/en/developer/packaging.rst @@ -545,7 +545,12 @@ would simply contain: Python-based packages --------------------- -The best way to install a Python-based package is to use pip, in which +Python-based packages should declare ``$(PYTHON)`` as a dependency, +and most Python-based packages will also have ``$(PYTHON_TOOLCHAIN)`` as +an order-only dependency, which will ensure that fundamental packages such +as ``pip`` and ``setuptools`` are available at the time of building the package. + +The best way to install a Python-based package is to use ``pip``, in which case the ``spkg-install.in`` script template might just consist of .. CODE-BLOCK:: bash @@ -556,19 +561,18 @@ Where ``sdh_pip_install`` is a function provided by ``sage-dist-helpers`` that points to the correct ``pip`` for the Python used by Sage, and includes some default flags needed for correct installation into Sage. -If pip will not work but a command like ``python3 setup.py install`` +If ``pip`` will not work for a package but a command like ``python3 setup.py install`` will, you may use ``sdh_setup_bdist_wheel``, followed by ``sdh_store_and_pip_install_wheel .``. -For ``spkg-check.in`` script templates, make sure to call -``sage-python23`` rather than ``python``. This will ensure that the -correct version of Python is used to check the package. -The same holds for ; for example, the ``scipy`` ``spkg-check.in`` -file contains the line +For ``spkg-check.in`` script templates, use ``python3`` rather +than just ``python``. The paths are set by the Sage build system +so that this runs the correct version of Python. +For example, the ``scipy`` ``spkg-check.in`` file contains the line .. CODE-BLOCK:: bash - exec sage-python23 spkg-check.py + exec python3 spkg-check.py All normal Python packages must have a file ``install-requires.txt``. If a Python package is available on PyPI, this file must contain the From 0327ca8adccf5bc475ac0f6ca1fd57f020c8ee45 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 18 Nov 2021 15:39:44 -0800 Subject: [PATCH 02/30] README.md: Update section Directory Layout --- README.md | 46 ++++++++++++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index f4d21b555c1..cf1c448835c 100644 --- a/README.md +++ b/README.md @@ -349,9 +349,9 @@ Simplified directory layout (only essential files/directories): SAGE_ROOT Root directory (sage-x.y.z in Sage tarball) ├── build │ └── pkgs Every package is a subdirectory here -│ ├── 4ti2 +│ ├── 4ti2/ │ … -│ └── zn_poly +│ └── zn_poly/ ├── configure Top-level configure script ├── COPYING.txt Copyright information ├── pkgs Source trees of Python distribution packages @@ -359,24 +359,40 @@ SAGE_ROOT Root directory (sage-x.y.z in Sage tarball) │ │ ├── sage_conf.py │ │ └── setup.py │ ├── sage-docbuild -│ │ ├── sage_docbuild +│ │ ├── sage_docbuild/ +│ │ └── setup.py +│ ├── sage-setup +│ │ ├── sage_setup/ │ │ └── setup.py │ ├── sage-sws2rst -│ │ ├── sage_sws2rst +│ │ ├── sage_sws2rst/ │ │ └── setup.py │ └── sagemath-standard -│ ├── bin -│ ├── sage +│ ├── bin/ +│ ├── sage -> ../../src/sage │ └── setup.py -├── local (SAGE_LOCAL) Compiled packages are installed here +├── local (SAGE_LOCAL) Installation hierarchy for non-Python packages │ ├── bin Executables │ ├── include C/C++ headers -│ ├── lib Shared libraries +│ ├── lib Shared libraries, architecture-dependent data │ ├── share Databases, architecture-independent data, docs │ │ └── doc Viewable docs of Sage and of some components │ └── var -│ ├── lib/sage List of installed packages -│ └── tmp/sage Temporary files when building Sage +│ ├── lib/sage +│ │ ├── installed/ +│ │ │ Records of installed non-Python packages +│ │ ├── scripts/ Scripts for uninstalling installed packages +│ │ └── venv-python3.9 (SAGE_VENV) +│ │ │ Installation hierarchy (virtual environment) +│ │ │ for Python packages +│ │ ├── bin/ Executables and installed scripts +│ │ ├── lib/python3.9/site-packages/ +│ │ │ Python modules/packages are installed here +│ │ └── var/lib/sage/ +│ │ └── wheels/ +│ │ Python wheels for all installed Python packages +│ │ +│ └── tmp/sage/ Temporary files when building Sage ├── logs │ ├── dochtml.log Log of the documentation build │ ├── install.log Full install log @@ -384,19 +400,21 @@ SAGE_ROOT Root directory (sage-x.y.z in Sage tarball) │ ├── alabaster-0.7.12.log │ … │ └── zn_poly-0.9.2.log -├── m4 M4 macros for configure +├── m4 M4 macros for generating the configure script │ └── *.m4 ├── Makefile Running "make" uses this file +├── prefix -> SAGE_LOCAL Convenience symlink to the installation tree ├── README.md This file ├── sage Script to start Sage ├── src Monolithic Sage library source tree -│ ├── bin Scripts that Sage uses internally -│ ├── doc Sage documentation sources -│ └── sage The Sage library source code +│ ├── bin/ Scripts that Sage uses internally +│ ├── doc/ Sage documentation sources +│ └── sage/ The Sage library source code ├── upstream Source tarballs of packages │ ├── Babel-2.9.1.tar.gz │ … │ └── zn_poly-0.9.2.tar.gz +├── venv -> SAGE_VENV Convenience symlink to the virtual environment └── VERSION.txt ``` For more details see [our Developer's Guide](https://doc.sagemath.org/html/en/developer/coding_basics.html#files-and-directory-structure). From 1d3a470b2370ecabd22692ac67d430fa0b37b13b Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 18 Nov 2021 19:06:26 -0800 Subject: [PATCH 03/30] src/doc/en/developer/python_packaging.rst: New --- src/doc/en/developer/index.rst | 1 + src/doc/en/developer/python_packaging.rst | 29 +++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 src/doc/en/developer/python_packaging.rst diff --git a/src/doc/en/developer/index.rst b/src/doc/en/developer/index.rst index 899ebe45a88..edd0fc5cc1c 100644 --- a/src/doc/en/developer/index.rst +++ b/src/doc/en/developer/index.rst @@ -159,6 +159,7 @@ Sage Coding Details coding_in_python coding_in_cython coding_in_other + python_packaging Packaging Third-Party Code -------------------------- diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst new file mode 100644 index 00000000000..a8eeed86c5e --- /dev/null +++ b/src/doc/en/developer/python_packaging.rst @@ -0,0 +1,29 @@ + +.. _chapter-modularization: + +=============================================== + Packaging Portions of the Sage Python Library +=============================================== + +The Sage Python library consists of a large number of Python modules, +organized into a hierarchical set of packages that fill the namespace +`sage`. All source files are located in a subdirectory of the +directory ``SAGE_ROOT/src/sage/``. + +For example, + +- the file ``SAGE_ROOT/src/sage/coding/code_bounds.py`` provides the + module ``sage.coding.code_bounds``; + +- the directory containing this file, ``SAGE_ROOT/src/sage/coding/``, + thus provides the package ``sage.coding``. This directory contains + an ``__init__.py`` file, which makes this package an "ordinary" + package. (Later, we will also discuss package directories without + ``__init__.py`` files, implicit (or "native") namespace packages.) + +There is another notion of "package" in Python, the "distribution +package" (also known as a "distribution" or a "pip-installable +package"). Currently, the entire Sage Python library is provided by a +single distribution, https://pypi.org/project/sagemath-standard/, +which is generated from the directory +``SAGE_ROOT/pkgs/sagemath-standard``. From 7606c55df95fec0599603573e025b40ad7c66daa Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 18 Nov 2021 19:43:00 -0800 Subject: [PATCH 04/30] src/doc/en/developer/python_packaging.rst: Document distribution package source layout --- src/doc/en/developer/python_packaging.rst | 38 ++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index a8eeed86c5e..4ea5f9e41b9 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -5,6 +5,10 @@ Packaging Portions of the Sage Python Library =============================================== + +Modules, packages, distribution packages +======================================== + The Sage Python library consists of a large number of Python modules, organized into a hierarchical set of packages that fill the namespace `sage`. All source files are located in a subdirectory of the @@ -19,7 +23,8 @@ For example, thus provides the package ``sage.coding``. This directory contains an ``__init__.py`` file, which makes this package an "ordinary" package. (Later, we will also discuss package directories without - ``__init__.py`` files, implicit (or "native") namespace packages.) + ``__init__.py`` files, which provide "implicit" (or "native") + "namespace" packages.) There is another notion of "package" in Python, the "distribution package" (also known as a "distribution" or a "pip-installable @@ -27,3 +32,34 @@ package"). Currently, the entire Sage Python library is provided by a single distribution, https://pypi.org/project/sagemath-standard/, which is generated from the directory ``SAGE_ROOT/pkgs/sagemath-standard``. + +Note that the distribution name is not required to be a Python +identifier. In fact, using dashes (``-``) is preferred to underscores in +distribution names; ``setuptools`` and other parts of Python's packaging +infrastructure normalize underscores to dashes. (Using dots in +distribution names, to indicate ownership by organizations, still +mentioned in https://www.python.org/dev/peps/pep-0423/, appears to +have largely fallen out of favor.) + + +Source directories of distribution packages +=========================================== + +The source directory of a distribution package, such as +``SAGE_ROOT/pkgs/sagemath-standard``, contains the following files: + +- ``sage`` -- a relative symbolic link to the monolithic Sage library + source tree ``SAGE_ROOT/src/sage/`` + +- ``MANIFEST.in`` -- controls which files and directories of the + monolithic Sage library source tree are included in the distribution + +- ``pyproject.toml.m4``, ``setup.cfg.m4``, ``requirements.txt.m4`` -- + Python packaging metadata, declaring the distribution name, dependencies, + etc. (These files are run through the ``m4`` macro processor by the + ``SAGE_ROOT/bootstrap`` script to generate standard files + ``pyproject.toml`` etc.) + +- ``setup.py`` -- a ``setuptools``-based installation script + +- ``tox.ini`` -- testing infrastructure From 70da678a1a94fdebe90f8523705395270e0d9960 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 18 Nov 2021 20:03:35 -0800 Subject: [PATCH 05/30] src/doc/en/developer/python_packaging.rst: Document dependencies --- src/doc/en/developer/python_packaging.rst | 88 +++++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 4ea5f9e41b9..45500cf793e 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -63,3 +63,91 @@ The source directory of a distribution package, such as - ``setup.py`` -- a ``setuptools``-based installation script - ``tox.ini`` -- testing infrastructure + + +Dependencies and distribution packages +====================================== + +When preparing a portion of the Sage library as a distribution +package, dependencies matter. + +Build-time dependencies +----------------------- + +If the portion of the library contains any Cython modules, these +modules are compiled during the wheel-building phase of the +distribution package. If the Cython module uses ``cimport`` to pull in +anything from ``.pxd`` files, these files must be either part of the +portion shipped as the distribution being built, or the distribution +that provides these files must be installed in the build +environment. Also, any C/C++ libraries that the Cython module uses +must be accessible from the build environment. + +Declaring build-time dependencies: Modern Python packaging provides a +mechanism to declare build-time dependencies on other distribution +packages via the file ``pyproject.toml`` ("build-system requires"); this +has superseded the older ``setup_requires`` declaration. (There is no +mechanism to declare anything regarding the C/C++ libraries.) + +Module-level runtime dependencies +--------------------------------- + +Any ``import`` statements at the top level of a Python or Cython +module are executed when the module is imported. Hence, the imported +modules must be part of the distribution, or provided by another +distribution -- which then must be declared as a run-time dependency. + +Declaring run-time dependencies: These dependencies are declared in +``setup.cfg`` (generated from ``setup.cfg.m4``) as ``install_requires``. + +Other runtime dependencies +-------------------------- + +If ``import`` statements are used within a method, the imported module +is loaded the first time that the method is called. Hence the module +defining the method can still be imported even if the module needed by +the method is not present. + +It is then a question whether a run-time dependency should be +declared. If the method needing that import provides core +functionality, then probably yes. But if it only provides what can be +considered "optional functionality", then probably not, and in this +case it will be up to the user to install the distribution enabling +this optional functionality. + +Declaring optional run-time dependencies: It is possible to declare +such optional dependencies as ``extra_requires`` in ``setup.cfg`` +(generated from ``setup.cfg.m4``). This is a very limited mechanism +-- in particular it does not affect the build phase of the +distribution in any way. It basically only provides a way to give a +nickname to a distribution that can be installed as an add-on. It +allows users to write, for example, ``pip install +sagemath-polyhedra[normaliz]`` instead of ``pip install +sagemath-polyhedra pynormaliz``. + +Lazy module-level imports +------------------------- + +Sage provides the ``lazy_import`` mechanism. Lazy imports can be +declared at the module level, but the actual importing is only done on +demand. It is a runtime error at that time if the imported module is +not present. This can be convenient compared to local imports in +methods when the same imports are needed in several methods. + +Doctest-only dependencies +------------------------- + +Doctests often use examples constructed using functionality provided +by other portions of the Sage library. This kind of integration +testing as one of the strengths of Sage; but it also creates extra +dependencies. + +Fortunately, these dependencies are very mild, and we can deal with +them using the same mechanism that we use for making doctests +conditional on the presence of optional libraries: using ``# optional - +FEATURE`` directives in the doctests. Adding these directives will +allow developers to test the distribution separately, without +requiring all of Sage to be present. + +Declaring doctest-only dependencies: The ``extra_requires`` mechanism +mentioned above can also be used for this. From 3f80c6e55ab4eafd5bb81821fde9c00a85c61ebd Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 18 Nov 2021 20:23:42 -0800 Subject: [PATCH 06/30] src/doc/en/developer/python_packaging.rst: Version constraints --- src/doc/en/developer/python_packaging.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 45500cf793e..ca84868f9f2 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -151,3 +151,13 @@ requiring all of Sage to be present. Declaring doctest-only dependencies: The ``extra_requires`` mechanism mentioned above can also be used for this. + + +Version constraints of dependencies +----------------------------------- + +The version information for dependencies comes from the files +``build/pkgs/*/install-requires.txt`` and +``build/pkgs/*/package-version.txt``. We use the ``m4`` macro +processor to insert the version information in the generated files +``pyproject.toml``, ``setup.cfg``, ``requirements.txt``. From 190ee43a071c0f7457e1b87a795f19ea23d8a824 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 14:39:40 -0800 Subject: [PATCH 07/30] src/doc/en/developer/python_packaging.rst: Add section on testing --- src/doc/en/developer/python_packaging.rst | 98 ++++++++++++++++++++++- 1 file changed, 97 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index ca84868f9f2..5bccabdda9e 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -35,7 +35,7 @@ which is generated from the directory Note that the distribution name is not required to be a Python identifier. In fact, using dashes (``-``) is preferred to underscores in -distribution names; ``setuptools`` and other parts of Python's packaging +distribution names; **setuptools** and other parts of Python's packaging infrastructure normalize underscores to dashes. (Using dots in distribution names, to indicate ownership by organizations, still mentioned in https://www.python.org/dev/peps/pep-0423/, appears to @@ -161,3 +161,99 @@ The version information for dependencies comes from the files ``build/pkgs/*/package-version.txt``. We use the ``m4`` macro processor to insert the version information in the generated files ``pyproject.toml``, ``setup.cfg``, ``requirements.txt``. + + +Testing distribution packages +============================= + +Of course, we need tools for testing modularized distributions of +portions of the Sage library. + +1. Modularized distributions must be testable separately! + +2. But we want to keep integration testing with other portions of Sage too! + +Preparing doctests +------------------ + +Enter ``# optional``, the doctest annotation that we also use whenever +an optional package is needed for a particular test. + +This mechanism can also be used for making a doctest conditional on +the presence of a portion of the Sage library. The available tags +take the form of package or module names such as ``sage.combinat``, +``sage.graphs``, ``sage.plot``, ``sage.rings.number_field``, +``sage.rings.real_double``, and ``sage.symbolic``. They are defined +via "features" in a single file, +``SAGE_ROOT/src/sage/features/sagemath.py``, which also provides the +mapping from features to the distributions providing them (actually, +to SPKG names). Using this mapping, Sage can issue installation hints +to the user. + +For example, the package ``sage.tensor`` is purely algebraic and has +no dependency on symbolics. However, there are a small number of +doctests that depend on the Symbolic Ring for integration +testing. Hence, these doctests are marked ``# optional - +sage.symbolic``. + +Testing the distribution in virtual environments with tox +--------------------------------------------------------- + +So how to test that this works? + +Sure, we could go into the installation directory +``SAGE_VENV/lib/python3.9/site-packages/`` and do ``rm -rf +sage/symbolic`` and test that things still work. But that's not a good +way of testing. + +Instead, we use a virtual environment in which we only install the +distribution to be tested (and its Python dependencies). + +Let's try it out first with the entire Sage library, represented by +the distribution **sagemath-standard**. Note that after Sage has been +built normally, a set of wheels for all installed Python packages is +available in ``SAGE_VENV/var/lib/sage/wheels/``:: + + $ ls venv/var/lib/sage/wheels + Babel-2.9.1-py2.py3-none-any.whl + Cython-0.29.24-cp39-cp39-macosx_11_0_x86_64.whl + Jinja2-2.11.2-py2.py3-none-any.whl + ... + sage_conf-9.5b6-py3-none-any.whl + ... + scipy-1.7.2-cp39-cp39-macosx_11_0_x86_64.whl + setuptools-58.2.0-py3-none-any.whl + ... + wheel-0.37.0-py2.py3-none-any.whl + widgetsnbextension-3.5.1-py2.py3-none-any.whl + zipp-3.5.0-py3-none-any.whl + +Note in particular the wheel for **sage-conf**, which provides +configuration variable settings and the connection to the non-Python +packages installed in ``SAGE_LOCAL``. + +We can now set up a separate virtual environment, in which we install +these wheels and our distribution to be tested. This is where ``tox`` +comes into play: It is the standard Python tool for creating +disposable virtual environments for testing. Every distribution in +``SAGE_ROOT/pkgs/`` provides a configuration file ``tox.ini``. + +Following the comments in the file +``SAGE_ROOT/pkgs/sagemath-standard/tox.ini``, we can try the following +command:: + + $ ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' + +This command does not make any changes to the normal installation of +Sage. The virtual environment is created in a subdirectory of +``pkgs/sagemath-standard-no-symbolics/.tox/``. After the command +finishes, we can start the separate installation of the Sage library +in its virtual environment:: + + $ pkgs/sagemath-standard/.tox/py39-sagewheels-nopypi/bin/sage + +We can also run parts of the testsuite:: + + $ pkgs/sagemath-standard/.tox/py39-sagewheels-nopypi/bin/sage -tp 4 src/sage/graphs/ + +The whole ``.tox`` directory can be safely deleted at any time. From 8da48f22febf923dd32bc4a1de0fa98443b14f2f Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 15:00:05 -0800 Subject: [PATCH 08/30] src/doc/en/developer/python_packaging.rst: More on dependencies --- src/doc/en/developer/python_packaging.rst | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 5bccabdda9e..f7c33e8ad18 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -83,12 +83,26 @@ that provides these files must be installed in the build environment. Also, any C/C++ libraries that the Cython module uses must be accessible from the build environment. -Declaring build-time dependencies: Modern Python packaging provides a +*Declaring build-time dependencies:* Modern Python packaging provides a mechanism to declare build-time dependencies on other distribution packages via the file ``pyproject.toml`` ("build-system requires"); this has superseded the older ``setup_requires`` declaration. (There is no mechanism to declare anything regarding the C/C++ libraries.) +While the namespace ``sage.*`` is organized roughly according to +mathematical fields or categories, how we partition the implementation +modules into distribution packages has to respect the hard constraints +that are imposed by the build-time dependencies. + +We can define some meaningful small distributions that just consist of +a single or a few Cython modules. For example, **sagemath-tdlib** +(https://trac.sagemath.org/ticket/29864) would just package the single +Cython module that must be linked with ``tdlib``, +:mod:`sage.graphs.graph_decompositions.tdlib`. Starting with the Sage +9.6 development cycle, as soon as namespace packages are activated, we +can start to create these distributions. This is quite a mechanical +task. + Module-level runtime dependencies --------------------------------- @@ -97,7 +111,7 @@ module are executed when the module is imported. Hence, the imported modules must be part of the distribution, or provided by another distribution -- which then must be declared as a run-time dependency. -Declaring run-time dependencies: These dependencies are declared in +*Declaring run-time dependencies:* These dependencies are declared in ``setup.cfg`` (generated from ``setup.cfg.m4``) as ``install_requires``. Other runtime dependencies @@ -115,7 +129,7 @@ considered "optional functionality", then probably not, and in this case it will be up to the user to install the distribution enabling this optional functionality. -Declaring optional run-time dependencies: It is possible to declare +*Declaring optional run-time dependencies:* It is possible to declare such optional dependencies as ``extra_requires`` in ``setup.cfg`` (generated from ``setup.cfg.m4``). This is a very limited mechanism -- in particular it does not affect the build phase of the @@ -149,7 +163,7 @@ FEATURE`` directives in the doctests. Adding these directives will allow developers to test the distribution separately, without requiring all of Sage to be present. -Declaring doctest-only dependencies: The ``extra_requires`` mechanism +*Declaring doctest-only dependencies:* The ``extra_requires`` mechanism mentioned above can also be used for this. From 5f99b19af8327f8a134f3d1db99a02e4f9f4c450 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 15:39:26 -0800 Subject: [PATCH 09/30] src/doc/en/developer/python_packaging.rst: Namespace packages --- src/doc/en/developer/python_packaging.rst | 52 ++++++++++++++++++++--- 1 file changed, 47 insertions(+), 5 deletions(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index f7c33e8ad18..4f5c732b6fb 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -11,7 +11,7 @@ Modules, packages, distribution packages The Sage Python library consists of a large number of Python modules, organized into a hierarchical set of packages that fill the namespace -`sage`. All source files are located in a subdirectory of the +``sage``. All source files are located in a subdirectory of the directory ``SAGE_ROOT/src/sage/``. For example, @@ -22,9 +22,7 @@ For example, - the directory containing this file, ``SAGE_ROOT/src/sage/coding/``, thus provides the package ``sage.coding``. This directory contains an ``__init__.py`` file, which makes this package an "ordinary" - package. (Later, we will also discuss package directories without - ``__init__.py`` files, which provide "implicit" (or "native") - "namespace" packages.) + package. There is another notion of "package" in Python, the "distribution package" (also known as a "distribution" or a "pip-installable @@ -41,6 +39,50 @@ distribution names, to indicate ownership by organizations, still mentioned in https://www.python.org/dev/peps/pep-0423/, appears to have largely fallen out of favor.) +Ordinary vs. implicit namespace packages +---------------------------------------- + +Each module of the Sage library must be packaged in exactly one +distribution package. + +An important constraint is that an "ordinary" package (directory with +``__init__.py`` file) cannot be split into more than one distribution +package. + +By removing the ``__init__.py`` file, however, we can make +the package an "implicit" (or "native") "namespace" package, following +https://www.python.org/dev/peps/pep-0420/; in this case, +``__init__.py`` cannot be used any more for initializing the package. + +Whenever there are two distribution packages that provide modules with +a common prefix of Python packages, that prefix needs to be a native +namespace package, i.e., there cannot be an ``__init__.py`` file. +For example, + +- **sagemath-tdlib** will provide ``sage.graphs.graph_decompositions.tdlib``, + +- **sagemath-rw** will provide ``sage.graphs.graph_decompositions.rankwidth``, + +- **sagemath-graphs** will provide all of the rest of + ``sage.graphs.graph_decompositions`` (and most of ``sage.graphs``). + +Then, none of + +- ``sage``, + +- ``sage.graphs``, + +- ``sage.graphs.graph_decomposition`` + +can be an ordinary package (with an ``__init__.py`` file), but rather +each of them has to be an implicit namespace package (no +``__init__.py`` file). + + +In the Sage 9.6 development cycle, several packages are converted to +implicit namespace packages. + + Source directories of distribution packages =========================================== @@ -60,7 +102,7 @@ The source directory of a distribution package, such as ``SAGE_ROOT/bootstrap`` script to generate standard files ``pyproject.toml`` etc.) -- ``setup.py`` -- a ``setuptools``-based installation script +- ``setup.py`` -- a **setuptools**-based installation script - ``tox.ini`` -- testing infrastructure From c7e56f6c185e667dd979c7a6efe4011cb828ddb7 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 15:59:59 -0800 Subject: [PATCH 10/30] src/doc/en/developer/python_packaging.rst: Explain monorepo --- src/doc/en/developer/python_packaging.rst | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 4f5c732b6fb..2d1c1c852b3 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -79,14 +79,25 @@ each of them has to be an implicit namespace package (no ``__init__.py`` file). -In the Sage 9.6 development cycle, several packages are converted to -implicit namespace packages. +In the Sage 9.6 development cycle, we still use ordinary packages by +default, but several packages are converted to implicit namespace +packages to support modularization. Source directories of distribution packages =========================================== +The development of the SageMath library uses a monorepo strategy for +all distribution packages that fill the `sage.*` namespace. This +means that the source trees of these distributions are included in a +single ``git`` repository, in a subdirectory of ``SAGE_ROOT/pkgs``. + +All these distribution packages have matching version numbers. From +the viewpoint of a single distribution, this means that sometimes +there will be a new release of some distribution where the only thing +changing is the version number. + The source directory of a distribution package, such as ``SAGE_ROOT/pkgs/sagemath-standard``, contains the following files: From 9a7096ee7965710b52e16354c46278bb648f13f4 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 16:02:15 -0800 Subject: [PATCH 11/30] src/doc/en/developer/python_packaging.rst: Explain symlinks --- src/doc/en/developer/python_packaging.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 2d1c1c852b3..f1b9e82042f 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -117,6 +117,10 @@ The source directory of a distribution package, such as - ``tox.ini`` -- testing infrastructure +The technique of using symbolic links pointing into ``SAGE_ROOT/src`` +has allowed the modularization effort to keep the ``SAGE_ROOT/src`` +tree monolithic: Modularization has been happening behind the scenes +and will not change where Sage developers find the source files. Dependencies and distribution packages ====================================== From ba0e4a870a546ad22e4b94e15c8a5b1f40bc365a Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 16:13:44 -0800 Subject: [PATCH 12/30] src/doc/en/developer/python_packaging.rst: Policy for distribution names --- src/doc/en/developer/python_packaging.rst | 35 +++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index f1b9e82042f..5efb7e58665 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -39,6 +39,41 @@ distribution names, to indicate ownership by organizations, still mentioned in https://www.python.org/dev/peps/pep-0423/, appears to have largely fallen out of favor.) +A distribution that provides Python modules in the ``sage.*`` namespace +(``sage.PAC.KAGE.MODULE``) should be named **sagemath-DISTRI-BUTION**. +Example: + +- The distribution https://pypi.org/project/sagemath-categories/ + provides a small subset of the modules of the Sage library, mostly + from the packages ``sage.structure``, ``sage.categories``, and + ``sage.misc``. + +Other distributions should not use the prefix **sagemath-** in the +distribution name. Example: + +- The distribution https://pypi.org/project/sage-sws2rst/ provides the + Python package ``sage_sws2rst``, so it does not fill the ``sage.*`` + namespace. + +A distribution that provides functionality that does not need to +import anything from the ``sage.*`` namespace should not use the +``sage.*`` namespace for its own packages/modules. It should be +positioned as part of the general Python ecosystem instead of as a +Sage-specific distribution. Examples: + +- The distribution https://pypi.org/project/pplpy/ provides the Python + package ``ppl`` and is a much extended version of what used to be + ``sage.libs.ppl``, a part of the Sage library. ``sage.libs.ppl`` had + dependencies on :mod:`sage.rings` to convert to/from Sage number + types. **pplpy** has no such dependencies and is therefore usable in a + wider range of Python projects. + +- The distribution https://pypi.org/project/memory-allocator/ provides + the Python package ``memory_allocator``. This used to be + ``sage.ext.memory_allocator``, a part of the Sage library. + + + Ordinary vs. implicit namespace packages ---------------------------------------- From 6c977790f7d180ff91b293254c21cde927fb68c8 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 23:07:19 -0800 Subject: [PATCH 13/30] src/doc/en/developer/python_packaging.rst: Add remarks on reducing dependencies --- src/doc/en/developer/python_packaging.rst | 32 +++++++++++++++++------ 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 5efb7e58665..e0d58a9d9f3 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -195,6 +195,13 @@ Cython module that must be linked with ``tdlib``, can start to create these distributions. This is quite a mechanical task. +*Reducing build-time dependencies:* Sometimes it is possible to +replace build-time dependencies of a Cython module on a library by a +runtime dependency. In other cases, it may be possible to split a +module that simultaneously depends on several libraries into smaller +modules, each of which has narrower dependencies. + + Module-level runtime dependencies --------------------------------- @@ -206,6 +213,23 @@ distribution -- which then must be declared as a run-time dependency. *Declaring run-time dependencies:* These dependencies are declared in ``setup.cfg`` (generated from ``setup.cfg.m4``) as ``install_requires``. +*Reducing module-level run-time dependencies:* + +- Avoid importing from ``sage.PAC.KAGE.all`` modules when + ``sage.PAC.KAGE`` is a namespace package. For example, no Sage + library code should import from ``sage.rings.all``. + +- Replace module-level imports by method-level imports. Note that + this comes with a small runtime overhead, which can become + noticeable if the method is called in tight inner loops. + +- Sage provides the ``lazy_import`` mechanism. Lazy imports can be + declared at the module level, but the actual importing is only done + on demand. It is a runtime error at that time if the imported module + is not present. This can be convenient compared to local imports in + methods when the same imports are needed in several methods. + + Other runtime dependencies -------------------------- @@ -231,14 +255,6 @@ allows users to write, for example, ``pip install sagemath-polyhedra[normaliz]`` instead of ``pip install sagemath-polyhedra pynormaliz``. -Lazy module-level imports -------------------------- - -Sage provides the ``lazy_import`` mechanism. Lazy imports can be -declared at the module level, but the actual importing is only done on -demand. It is a runtime error at that time if the imported module is -not present. This can be convenient compared to local imports in -methods when the same imports are needed in several methods. Doctest-only dependencies ------------------------- From 4eb58daac24a3dd408a151b65635a6889faf17f2 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 23:30:58 -0800 Subject: [PATCH 14/30] src/doc/en/developer/python_packaging.rst: More on testing --- src/doc/en/developer/python_packaging.rst | 25 ++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index e0d58a9d9f3..02b1b8416de 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -261,7 +261,7 @@ Doctest-only dependencies Doctests often use examples constructed using functionality provided by other portions of the Sage library. This kind of integration -testing as one of the strengths of Sage; but it also creates extra +testing is one of the strengths of Sage; but it also creates extra dependencies. Fortunately, these dependencies are very mild, and we can deal with @@ -379,3 +379,26 @@ We can also run parts of the testsuite:: $ pkgs/sagemath-standard/.tox/py39-sagewheels-nopypi/bin/sage -tp 4 src/sage/graphs/ The whole ``.tox`` directory can be safely deleted at any time. + +We can do the same with other distributions, for example the large +distribution **sagemath-standard-no-symbolics** +(https://trac.sagemath.org/ticket/32601), which is intended to provide +everything that is currently in the standard Sage library, i.e., +without depending on optional packages, but without the packages +``sage.symbolic``, ``sage.functions``, ``sage.calculus``, etc. + +Again we can run the test with ``tox`` in a separate virtual environment:: + + $ ./sage -sh -c '(cd pkgs/sagemath-standard-no-symbolics && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' + +Some small distributions, for example the ones providing the two +lowest levels, **sagemath-objects** and **sagemath-categories** +(https://trac.sagemath.org/ticket/29865), can be installed and tested +without relying on the wheels from the Sage build:: + + $ ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -v -v -v -e py39)' + +This command finds the declared build-time and run-time dependencies +on PyPI, either as source tarballs or as prebuilt wheels, and builds +and installs the distribution **sagemath-objects** in a virtual +environment in a subdirectory of ``pkgs/sagemath-objects/.tox``. From c118c881e11f90750407920e3820ddfd59ccc77e Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 23:45:48 -0800 Subject: [PATCH 15/30] src/doc/en/developer/python_packaging.rst: More on reducing dependencies --- src/doc/en/developer/python_packaging.rst | 48 +++++++++++++++++++++-- 1 file changed, 44 insertions(+), 4 deletions(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 02b1b8416de..9a64b16fa95 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -245,15 +245,55 @@ considered "optional functionality", then probably not, and in this case it will be up to the user to install the distribution enabling this optional functionality. +As an example, let us consider designing a distribution that centers +around the package :mod:`sage.coding`. First, let's see if it uses symbolics:: + + (9.5.beta6) $ git grep -E 'sage[.](symbolic|functions|calculus)' src/sage/coding + src/sage/coding/code_bounds.py: from sage.functions.other import ceil + ... + src/sage/coding/grs_code.py:from sage.symbolic.ring import SR + ... + src/sage/coding/guruswami_sudan/utils.py:from sage.functions.other import floor + +Apparently it does not in a very substantial way: + +- The imports of the symbolic functions ``ceil`` and ``floor`` can + likely be replaced by :func:`~sage.arith.misc.integer_floor` and + :func:`~sage.arith.misc.integer_ceil`. + +- Looking at the import of ``SR`` by :mod:`sage.coding.grs_code`, it + seems that ``SR`` is used for running some symbolic sum, but the + doctests do not show symbolic results, so it is likely that this can + be replaced. + +- Note though that the above textual search for the module names is + merely a heuristic. Looking at the source of "entropy", through + ``log`` from :mod:`sage.misc.functional`, a runtime dependency on + symbolics comes in. In fact, for this reason, two doctests (I have + already marked 2 doctests there as # optional - sage.symbolic.) + +So if packaged as **sagemath-coding**, now a domain expert would have +to decide whether these dependencies on symbolics are strong enough to +declare a runtime dependency (``install_requires``) on +**sagemath-symbolics**. This declaration would mean that any user who +installs **sagemath-coding** (``pip install sagemath-coding``) would +pull in **sagemath-symbolics**, which has heavy compile-time +dependencies (ECL/Maxima/FLINT/Singular/...). + +The alternative is to consider the use of symbolics by +**sagemath-coding** merely as something that provides some extra +features ... which will only be working if the user also has installed +**sagemath-symbolics**. + *Declaring optional run-time dependencies:* It is possible to declare such optional dependencies as ``extra_requires`` in ``setup.cfg`` (generated from ``setup.cfg.m4``). This is a very limited mechanism -- in particular it does not affect the build phase of the distribution in any way. It basically only provides a way to give a -nickname to a distribution that can be installed as an add-on. It -allows users to write, for example, ``pip install -sagemath-polyhedra[normaliz]`` instead of ``pip install -sagemath-polyhedra pynormaliz``. +nickname to a distribution that can be installed as an add-on. + +In our example, we could declare an ``extra_requires`` so that users +could use ``pip install sagemath-coding[symbolics]``. Doctest-only dependencies From e9a4cd5a94de35495d350bee8d9b54021e89bc64 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 23:51:30 -0800 Subject: [PATCH 16/30] src/doc/en/developer/python_packaging.rst: On testing small distributions --- src/doc/en/developer/python_packaging.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 9a64b16fa95..2ee341a0f9c 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -442,3 +442,9 @@ This command finds the declared build-time and run-time dependencies on PyPI, either as source tarballs or as prebuilt wheels, and builds and installs the distribution **sagemath-objects** in a virtual environment in a subdirectory of ``pkgs/sagemath-objects/.tox``. + +Building these small distributions serves as a valuable regression +testsuite. However, a current issue with both of these distributions +is that they are not separately testable: The doctests for these +modules depend on a lot of other functionality from higher-level parts +of the Sage library. From e8d1b950fa2c121e1b3f48517832464db5947361 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 21 Nov 2021 23:59:56 -0800 Subject: [PATCH 17/30] src/doc/en/developer/python_packaging.rst: Small edits --- src/doc/en/developer/python_packaging.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/python_packaging.rst index 2ee341a0f9c..2e70b6530b8 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/python_packaging.rst @@ -24,8 +24,8 @@ For example, an ``__init__.py`` file, which makes this package an "ordinary" package. -There is another notion of "package" in Python, the "distribution -package" (also known as a "distribution" or a "pip-installable +There is another notion of "package" in Python, the **distribution +package** (also known as a "distribution" or a "pip-installable package"). Currently, the entire Sage Python library is provided by a single distribution, https://pypi.org/project/sagemath-standard/, which is generated from the directory @@ -37,7 +37,8 @@ distribution names; **setuptools** and other parts of Python's packaging infrastructure normalize underscores to dashes. (Using dots in distribution names, to indicate ownership by organizations, still mentioned in https://www.python.org/dev/peps/pep-0423/, appears to -have largely fallen out of favor.) +have largely fallen out of favor, and we will not use it in the SageMath +project.) A distribution that provides Python modules in the ``sage.*`` namespace (``sage.PAC.KAGE.MODULE``) should be named **sagemath-DISTRI-BUTION**. From a4db1644dd0d9b9c27326012c7a24cd7d102ae22 Mon Sep 17 00:00:00 2001 From: Kwankyu Lee Date: Mon, 22 Nov 2021 18:12:33 +0900 Subject: [PATCH 18/30] More edits --- src/doc/en/developer/index.rst | 9 +++- ...ckaging.rst => packaging_sage_library.rst} | 54 +++++++++---------- 2 files changed, 34 insertions(+), 29 deletions(-) rename src/doc/en/developer/{python_packaging.rst => packaging_sage_library.rst} (92%) diff --git a/src/doc/en/developer/index.rst b/src/doc/en/developer/index.rst index edd0fc5cc1c..38074743daa 100644 --- a/src/doc/en/developer/index.rst +++ b/src/doc/en/developer/index.rst @@ -159,7 +159,14 @@ Sage Coding Details coding_in_python coding_in_cython coding_in_other - python_packaging + +Packaging the Sage Library +-------------------------- + +.. toctree:: + :maxdepth: 3 + + packaging_sage_library Packaging Third-Party Code -------------------------- diff --git a/src/doc/en/developer/python_packaging.rst b/src/doc/en/developer/packaging_sage_library.rst similarity index 92% rename from src/doc/en/developer/python_packaging.rst rename to src/doc/en/developer/packaging_sage_library.rst index 2e70b6530b8..7120ddf10d0 100644 --- a/src/doc/en/developer/python_packaging.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -1,15 +1,15 @@ .. _chapter-modularization: -=============================================== - Packaging Portions of the Sage Python Library -=============================================== +============================ + Packaging the Sage Library +============================ Modules, packages, distribution packages ======================================== -The Sage Python library consists of a large number of Python modules, +The Sage library consists of a large number of Python modules, organized into a hierarchical set of packages that fill the namespace ``sage``. All source files are located in a subdirectory of the directory ``SAGE_ROOT/src/sage/``. @@ -17,16 +17,14 @@ directory ``SAGE_ROOT/src/sage/``. For example, - the file ``SAGE_ROOT/src/sage/coding/code_bounds.py`` provides the - module ``sage.coding.code_bounds``; + module ``sage.coding.code_bounds``. - the directory containing this file, ``SAGE_ROOT/src/sage/coding/``, - thus provides the package ``sage.coding``. This directory contains - an ``__init__.py`` file, which makes this package an "ordinary" - package. + thus provides the package ``sage.coding``. There is another notion of "package" in Python, the **distribution package** (also known as a "distribution" or a "pip-installable -package"). Currently, the entire Sage Python library is provided by a +package"). Currently, the entire Sage library is provided by a single distribution, https://pypi.org/project/sagemath-standard/, which is generated from the directory ``SAGE_ROOT/pkgs/sagemath-standard``. @@ -64,7 +62,7 @@ Sage-specific distribution. Examples: - The distribution https://pypi.org/project/pplpy/ provides the Python package ``ppl`` and is a much extended version of what used to be - ``sage.libs.ppl``, a part of the Sage library. ``sage.libs.ppl`` had + ``sage.libs.ppl``, a part of the Sage library. The package ``sage.libs.ppl`` had dependencies on :mod:`sage.rings` to convert to/from Sage number types. **pplpy** has no such dependencies and is therefore usable in a wider range of Python projects. @@ -74,25 +72,23 @@ Sage-specific distribution. Examples: ``sage.ext.memory_allocator``, a part of the Sage library. +Ordinary packages vs. implicit namespace packages +------------------------------------------------- -Ordinary vs. implicit namespace packages ----------------------------------------- +Each module of the Sage library must be packaged in exactly one distribution +package. However, modules in a package may be included in different +distribution packages. In this regard, there is an important constaint that an +ordinary package (directory with ``__init__.py`` file) cannot be split into +more than one distribution package. -Each module of the Sage library must be packaged in exactly one -distribution package. +By removing the ``__init__.py`` file, however, we can make the package an +"implicit" (or "native") "namespace" package, following +https://www.python.org/dev/peps/pep-0420/. Implicit namespace packages can be +included in more than one distribution package. Hence whenever there are two +distribution packages that provide modules with a common prefix of Python +packages, that prefix needs to be a implicit namespace package, i.e., there +cannot be an ``__init__.py`` file. -An important constraint is that an "ordinary" package (directory with -``__init__.py`` file) cannot be split into more than one distribution -package. - -By removing the ``__init__.py`` file, however, we can make -the package an "implicit" (or "native") "namespace" package, following -https://www.python.org/dev/peps/pep-0420/; in this case, -``__init__.py`` cannot be used any more for initializing the package. - -Whenever there are two distribution packages that provide modules with -a common prefix of Python packages, that prefix needs to be a native -namespace package, i.e., there cannot be an ``__init__.py`` file. For example, - **sagemath-tdlib** will provide ``sage.graphs.graph_decompositions.tdlib``, @@ -114,6 +110,8 @@ can be an ordinary package (with an ``__init__.py`` file), but rather each of them has to be an implicit namespace package (no ``__init__.py`` file). +For an implicit namespace package, ``__init__.py`` cannot be used any more for +initializing the package. In the Sage 9.6 development cycle, we still use ordinary packages by default, but several packages are converted to implicit namespace @@ -124,8 +122,8 @@ packages to support modularization. Source directories of distribution packages =========================================== -The development of the SageMath library uses a monorepo strategy for -all distribution packages that fill the `sage.*` namespace. This +The development of the Sage library uses a monorepo strategy for +all distribution packages that fill the ``sage.*`` namespace. This means that the source trees of these distributions are included in a single ``git`` repository, in a subdirectory of ``SAGE_ROOT/pkgs``. From 4b4d6d3684c438d096e1985e6767f636b4db579e Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 22 Nov 2021 15:24:28 -0800 Subject: [PATCH 19/30] src/doc/en/developer/packaging_sage_library.rst: Explain ABCs --- .../en/developer/packaging_sage_library.rst | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 7120ddf10d0..17e372e5417 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -215,8 +215,10 @@ distribution -- which then must be declared as a run-time dependency. *Reducing module-level run-time dependencies:* - Avoid importing from ``sage.PAC.KAGE.all`` modules when - ``sage.PAC.KAGE`` is a namespace package. For example, no Sage - library code should import from ``sage.rings.all``. + ``sage.PAC.KAGE`` is a namespace package. The main purpose of the + ``*.all`` modules is for populating the global interactive + environment that is available to users at the ``sage:`` prompt. For + example, no Sage library code should import from ``sage.rings.all``. - Replace module-level imports by method-level imports. Note that this comes with a small runtime overhead, which can become @@ -228,6 +230,23 @@ distribution -- which then must be declared as a run-time dependency. is not present. This can be convenient compared to local imports in methods when the same imports are needed in several methods. +- Avoid the "modularization anti-pattern" of importing a class from + another module just to run an ``isinstance(object, Class)`` test, in + particular when the module implementing ``Class`` has heavy + dependencies. For example, importing the class + :class:`~sage.rings.padics.generic_nodes.pAdicField` (or the + function :class:`~sage.rings.padics.generic_nodes.is_pAdicField`) + requires the libraries NTL and PARI. + + Instead, provide an abstract base class (ABC) in a module that only + has light dependencies, make ``Class`` a subclass of ``ABC``, and + use ``isinstance(object, ABC)``. For example, :mod:`sage.rings.abc` + provides abstract base classes for many ring (parent) classes, + including :class:`sage.rings.abc.pAdicField`. So we can replace + ``isinstance(object, sage.rings.padics.generic_nodes.pAdicField)`` + and ``sage.rings.padics.generic_nodes.is_pAdicField(object)`` by + ``isinstance(object, sage.rings.abc.pAdicField)``. + Other runtime dependencies -------------------------- From 25fb88986b16968c2306443d719e6231db603aee Mon Sep 17 00:00:00 2001 From: Kwankyu Lee Date: Tue, 23 Nov 2021 10:41:27 +0900 Subject: [PATCH 20/30] Still more edits --- .../en/developer/packaging_sage_library.rst | 55 +++++++++---------- 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 17e372e5417..b1d649d9f0e 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -17,7 +17,7 @@ directory ``SAGE_ROOT/src/sage/``. For example, - the file ``SAGE_ROOT/src/sage/coding/code_bounds.py`` provides the - module ``sage.coding.code_bounds``. + module ``sage.coding.code_bounds``; - the directory containing this file, ``SAGE_ROOT/src/sage/coding/``, thus provides the package ``sage.coding``. @@ -38,8 +38,8 @@ mentioned in https://www.python.org/dev/peps/pep-0423/, appears to have largely fallen out of favor, and we will not use it in the SageMath project.) -A distribution that provides Python modules in the ``sage.*`` namespace -(``sage.PAC.KAGE.MODULE``) should be named **sagemath-DISTRI-BUTION**. +A distribution that provides Python modules in the ``sage.*`` namespace, say +mainly from ``sage.PAC.KAGE``, should be named **sagemath-DISTRI-BUTION**. Example: - The distribution https://pypi.org/project/sagemath-categories/ @@ -77,7 +77,7 @@ Ordinary packages vs. implicit namespace packages Each module of the Sage library must be packaged in exactly one distribution package. However, modules in a package may be included in different -distribution packages. In this regard, there is an important constaint that an +distribution packages. In this regard, there is an important constraint that an ordinary package (directory with ``__init__.py`` file) cannot be split into more than one distribution package. @@ -118,7 +118,6 @@ default, but several packages are converted to implicit namespace packages to support modularization. - Source directories of distribution packages =========================================== @@ -156,12 +155,14 @@ has allowed the modularization effort to keep the ``SAGE_ROOT/src`` tree monolithic: Modularization has been happening behind the scenes and will not change where Sage developers find the source files. + Dependencies and distribution packages ====================================== When preparing a portion of the Sage library as a distribution package, dependencies matter. + Build-time dependencies ----------------------- @@ -214,11 +215,11 @@ distribution -- which then must be declared as a run-time dependency. *Reducing module-level run-time dependencies:* -- Avoid importing from ``sage.PAC.KAGE.all`` modules when - ``sage.PAC.KAGE`` is a namespace package. The main purpose of the - ``*.all`` modules is for populating the global interactive - environment that is available to users at the ``sage:`` prompt. For - example, no Sage library code should import from ``sage.rings.all``. +- Avoid importing from ``sage.PAC.KAGE.all`` modules when ``sage.PAC.KAGE`` is + a namespace package. The main purpose of the ``*.all`` modules is for + populating the global interactive environment that is available to users at + the ``sage:`` prompt. In particular, no Sage library code should import from + ``sage.rings.all``. - Replace module-level imports by method-level imports. Note that this comes with a small runtime overhead, which can become @@ -287,8 +288,8 @@ Apparently it does not in a very substantial way: - Note though that the above textual search for the module names is merely a heuristic. Looking at the source of "entropy", through ``log`` from :mod:`sage.misc.functional`, a runtime dependency on - symbolics comes in. In fact, for this reason, two doctests (I have - already marked 2 doctests there as # optional - sage.symbolic.) + symbolics comes in. In fact, for this reason, two doctests there are + already marked as ``# optional - sage.symbolic``. So if packaged as **sagemath-coding**, now a domain expert would have to decide whether these dependencies on symbolics are strong enough to @@ -300,7 +301,7 @@ dependencies (ECL/Maxima/FLINT/Singular/...). The alternative is to consider the use of symbolics by **sagemath-coding** merely as something that provides some extra -features ... which will only be working if the user also has installed +features, which will only be working if the user also has installed **sagemath-symbolics**. *Declaring optional run-time dependencies:* It is possible to declare @@ -349,26 +350,24 @@ Testing distribution packages Of course, we need tools for testing modularized distributions of portions of the Sage library. -1. Modularized distributions must be testable separately! +- Modularized distributions must be testable separately! -2. But we want to keep integration testing with other portions of Sage too! +- But we want to keep integration testing with other portions of Sage too! Preparing doctests ------------------ -Enter ``# optional``, the doctest annotation that we also use whenever -an optional package is needed for a particular test. - -This mechanism can also be used for making a doctest conditional on -the presence of a portion of the Sage library. The available tags -take the form of package or module names such as ``sage.combinat``, -``sage.graphs``, ``sage.plot``, ``sage.rings.number_field``, -``sage.rings.real_double``, and ``sage.symbolic``. They are defined -via "features" in a single file, -``SAGE_ROOT/src/sage/features/sagemath.py``, which also provides the -mapping from features to the distributions providing them (actually, -to SPKG names). Using this mapping, Sage can issue installation hints -to the user. +Whenever an optional package is needed for a particular test, we use the +doctest annotation ``# optional``. This mechanism can also be used for making a +doctest conditional on the presence of a portion of the Sage library. + +The available tags take the form of package or module names such as +``sage.combinat``, ``sage.graphs``, ``sage.plot``, ``sage.rings.number_field``, +``sage.rings.real_double``, and ``sage.symbolic``. They are defined via +"features" in a single file, ``SAGE_ROOT/src/sage/features/sagemath.py``, which +also provides the mapping from features to the distributions providing them +(actually, to SPKG names). Using this mapping, Sage can issue installation +hints to the user. For example, the package ``sage.tensor`` is purely algebraic and has no dependency on symbolics. However, there are a small number of From 0955139476b14d3c2ed428f30cbacc3df5e6497c Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 22 Nov 2021 18:14:44 -0800 Subject: [PATCH 21/30] src/doc/en/developer/packaging_sage_library.rst: Add plot --- .../en/developer/packaging_sage_library.rst | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index b1d649d9f0e..69ba2d23c25 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -344,6 +344,30 @@ processor to insert the version information in the generated files ``pyproject.toml``, ``setup.cfg``, ``requirements.txt``. +Hierarchy of distribution packages +---------------------------------- + +.. PLOT:: + + dists = ['sagemath-objects', 'sagemath-categories', + 'sagemath-polyhedra', 'sagemath-standard-no-symbolics', + 'sagemath-singular', 'sagemath-symbolics', + 'sagemath-standard', 'sagemath-graphs', 'sagemath-tdlib'] + covers = [['sagemath-objects','sagemath-categories'], + ['sagemath-categories', 'sagemath-polyhedra'], + ['sagemath-categories', 'sagemath-graphs'], + ['sagemath-categories', 'sagemath-singular'], + ['sagemath-graphs', 'sagemath-standard-no-symbolics'], + ['sagemath-graphs', 'sagemath-tdlib'], + ['sagemath-polyhedra', 'sagemath-standard-no-symbolics'], + ['sagemath-standard-no-symbolics', 'sagemath-standard'], + ['sagemath-singular', 'sagemath-symbolics'], + ['sagemath-symbolics', 'sagemath-standard']] + P = Poset((dists,covers)) + g = P.plot(element_size=1000, element_shape="s") + sphinx_plot(g, figsize=(8, 4)) + + Testing distribution packages ============================= From e5ad5186a9835a8a82e1d6f25ec318ac9bb5c9f8 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 22 Nov 2021 19:15:39 -0800 Subject: [PATCH 22/30] pkgs/sagemath-standard/setup.py: Do not depend on sage_setup for sdist or egg_info --- pkgs/sagemath-standard/setup.py | 65 +++++++++++++++++---------------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/pkgs/sagemath-standard/setup.py b/pkgs/sagemath-standard/setup.py index 68844291f58..b94f197f014 100755 --- a/pkgs/sagemath-standard/setup.py +++ b/pkgs/sagemath-standard/setup.py @@ -28,21 +28,34 @@ sage.env.SAGE_SRC = os.getcwd() from sage.env import * -from sage_setup.excepthook import excepthook -sys.excepthook = excepthook - -from sage_setup.setenv import setenv -setenv() - ######################################################### ### Configuration ######################################################### -if len(sys.argv) > 1 and sys.argv[1] == "sdist": +if len(sys.argv) > 1 and (sys.argv[1] == "sdist" or sys.argv[1] == "egg_info"): sdist = True else: sdist = False +if sdist: + cmdclass = {} +else: + from sage_setup.excepthook import excepthook + sys.excepthook = excepthook + + from sage_setup.setenv import setenv + setenv() + + from sage_setup.command.sage_build import sage_build + from sage_setup.command.sage_build_cython import sage_build_cython + from sage_setup.command.sage_build_ext import sage_build_ext + from sage_setup.command.sage_install import sage_install_and_clean + + cmdclass = dict(build=sage_build, + build_cython=sage_build_cython, + build_ext=sage_build_ext, + install=sage_install_and_clean) + ######################################################### ### Testing related stuff ######################################################### @@ -52,27 +65,22 @@ if os.path.exists(sage.misc.lazy_import_cache.get_cache_file()): os.unlink(sage.misc.lazy_import_cache.get_cache_file()) - -from sage_setup.command.sage_build import sage_build -from sage_setup.command.sage_build_cython import sage_build_cython -from sage_setup.command.sage_build_ext import sage_build_ext - - ######################################################### ### Discovering Sources ######################################################### -# TODO: This should be quiet by default -print("Discovering Python/Cython source code....") -t = time.time() - -from sage_setup.optional_extension import is_package_installed_and_updated - if sdist: # No need to compute distributions. This avoids a dependency on Cython # just to make an sdist. distributions = None + python_packages = [] + python_modules = [] + cython_modules = [] else: + # TODO: This should be quiet by default + print("Discovering Python/Cython source code....") + t = time.time() + from sage_setup.optional_extension import is_package_installed_and_updated distributions = [''] optional_packages_with_extensions = ['mcqd', 'bliss', 'tdlib', 'primecount', 'coxeter3', 'fes', 'sirocco', 'meataxe'] @@ -80,17 +88,13 @@ for pkg in optional_packages_with_extensions if is_package_installed_and_updated(pkg)] log.warn('distributions = {0}'.format(distributions)) + from sage_setup.find import find_python_sources + python_packages, python_modules, cython_modules = find_python_sources( + SAGE_SRC, ['sage'], distributions=distributions) -from sage_setup.find import find_python_sources -python_packages, python_modules, cython_modules = find_python_sources( - SAGE_SRC, ['sage'], distributions=distributions) - -log.debug('python_packages = {0}'.format(python_packages)) - -print("Discovered Python/Cython sources, time: %.2f seconds." % (time.time() - t)) - + log.debug('python_packages = {0}'.format(python_packages)) + print("Discovered Python/Cython sources, time: %.2f seconds." % (time.time() - t)) -from sage_setup.command.sage_install import sage_install_and_clean ######################################################### ### Distutils @@ -175,8 +179,5 @@ 'bin/sage-update-src', 'bin/sage-update-version', ], - cmdclass = dict(build=sage_build, - build_cython=sage_build_cython, - build_ext=sage_build_ext, - install=sage_install_and_clean), + cmdclass = cmdclass, ext_modules = cython_modules) From 821d16e64c9506f33fcc57992bfe77512d707231 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 22 Nov 2021 19:19:03 -0800 Subject: [PATCH 23/30] src/doc/en/developer/packaging_sage_library.rst: Add bootstrap to testing instructions to avoid the trap of #32868 --- src/doc/en/developer/packaging_sage_library.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 69ba2d23c25..e8bf6e0c476 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -445,7 +445,7 @@ Following the comments in the file ``SAGE_ROOT/pkgs/sagemath-standard/tox.ini``, we can try the following command:: - $ ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' + $ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' This command does not make any changes to the normal installation of Sage. The virtual environment is created in a subdirectory of @@ -470,14 +470,14 @@ without depending on optional packages, but without the packages Again we can run the test with ``tox`` in a separate virtual environment:: - $ ./sage -sh -c '(cd pkgs/sagemath-standard-no-symbolics && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' + $ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard-no-symbolics && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' Some small distributions, for example the ones providing the two lowest levels, **sagemath-objects** and **sagemath-categories** (https://trac.sagemath.org/ticket/29865), can be installed and tested without relying on the wheels from the Sage build:: - $ ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -v -v -v -e py39)' + $ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -v -v -v -e py39)' This command finds the declared build-time and run-time dependencies on PyPI, either as source tarballs or as prebuilt wheels, and builds From 77c957f44881baa2f9b9c9a41a94098d9b9a0d7d Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Mon, 22 Nov 2021 19:37:08 -0800 Subject: [PATCH 24/30] src/doc/en/developer/packaging_sage_library.rst: Put hierarchy section one level higher --- src/doc/en/developer/packaging_sage_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index e8bf6e0c476..008417ba1cd 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -345,7 +345,7 @@ processor to insert the version information in the generated files Hierarchy of distribution packages ----------------------------------- +================================== .. PLOT:: From 9375ef85a66d83c62fe9658c7e99beff873db651 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 1 Oct 2021 18:57:58 -0700 Subject: [PATCH 25/30] pkgs/sagemath-standard/tox.ini: Use SAGE_VENV or venv symlink to find wheels --- pkgs/sagemath-standard/tox.ini | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/pkgs/sagemath-standard/tox.ini b/pkgs/sagemath-standard/tox.ini index f9ac7336547..5fa331ae3c8 100644 --- a/pkgs/sagemath-standard/tox.ini +++ b/pkgs/sagemath-standard/tox.ini @@ -25,7 +25,7 @@ envlist = # Build dependencies according to requirements.txt (all versions fixed). # Use ONLY the wheels built and stored by the Sage distribution (no PyPI): # - # ./sage -sh -c '(cd pkgs/sagelib && tox -v -v -v -e python-sagewheels-nopypi)' + # ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e python-sagewheels-nopypi)' # python-sagewheels-nopypi, # @@ -33,7 +33,7 @@ envlist = # using the dependencies declared in pyproject.toml and setup.cfg (install-requires) only: # Still use ONLY the wheels built and stored by the Sage distribution (no PyPI). # - # ./sage -sh -c '(cd pkgs/sagelib && tox -v -v -v -e python-sagewheels-nopypi-norequirements)' + # ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e python-sagewheels-nopypi-norequirements)' # python-sagewheels-nopypi-norequirements, # @@ -44,7 +44,7 @@ envlist = # and additionally allow packages from PyPI. # Because all versions are fixed, we "should" end up using the prebuilt wheels. # - # ./sage -sh -c '(cd pkgs/sagelib && tox -v -v -v -e python-sagewheels)' + # ./sage -sh -c '(cd pkgs/sagemath-standard && tox -v -v -v -e python-sagewheels)' # python-sagewheels, # @@ -81,8 +81,8 @@ passenv = PKG_CONFIG_PATH # Parallel build SAGE_NUM_THREADS - # SAGE_LOCAL only for finding the wheels - sagewheels: SAGE_LOCAL + # SAGE_VENV only for finding the wheels + sagewheels: SAGE_VENV # Location of the wheels (needs to include a PEP 503 compliant # Simple Repository index, i.e., a subdirectory "simple") sagewheels: SAGE_SPKG_WHEELS @@ -92,7 +92,7 @@ setenv = HOME={envdir} # We supply pip options by environment variables so that they # apply both to the installation of the dependencies and of the package - sagewheels: PIP_FIND_LINKS=file://{env:SAGE_SPKG_WHEELS:{env:SAGE_LOCAL:{toxinidir}/../../../../local}/var/lib/sage/wheels} + sagewheels: PIP_FIND_LINKS=file://{env:SAGE_SPKG_WHEELS:{env:SAGE_VENV:{toxinidir}/../../../../venv}/var/lib/sage/wheels} nopypi: PIP_NO_INDEX=true # No build isolation for PEP 517 packages - use what is already in the environment # Note that this pip env "NO" variable uses inverted logic: From c6fc11169fb777a699b6c0dd7d99c6197eb9a1f9 Mon Sep 17 00:00:00 2001 From: Kwankyu Lee Date: Tue, 23 Nov 2021 22:59:35 +0900 Subject: [PATCH 26/30] Prettier diagram --- .../en/developer/packaging_sage_library.rst | 31 +++++++++---------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 008417ba1cd..2dd5eacedc4 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -349,23 +349,20 @@ Hierarchy of distribution packages .. PLOT:: - dists = ['sagemath-objects', 'sagemath-categories', - 'sagemath-polyhedra', 'sagemath-standard-no-symbolics', - 'sagemath-singular', 'sagemath-symbolics', - 'sagemath-standard', 'sagemath-graphs', 'sagemath-tdlib'] - covers = [['sagemath-objects','sagemath-categories'], - ['sagemath-categories', 'sagemath-polyhedra'], - ['sagemath-categories', 'sagemath-graphs'], - ['sagemath-categories', 'sagemath-singular'], - ['sagemath-graphs', 'sagemath-standard-no-symbolics'], - ['sagemath-graphs', 'sagemath-tdlib'], - ['sagemath-polyhedra', 'sagemath-standard-no-symbolics'], - ['sagemath-standard-no-symbolics', 'sagemath-standard'], - ['sagemath-singular', 'sagemath-symbolics'], - ['sagemath-symbolics', 'sagemath-standard']] - P = Poset((dists,covers)) - g = P.plot(element_size=1000, element_shape="s") - sphinx_plot(g, figsize=(8, 4)) + def node(label, pos): + return text(label, (3*pos[0],2*pos[1]), background_color='pink', color='black') + def edge(start, end): + return arrow((3*start[0],2*start[1]+.5),(3*end[0],2*end[1]-.5), arrowsize=2) + g = Graphics() + g += (node("sagemath-objects", (1,0)) + edge((1,0),(1,1))) + g += (node("sagemath-categories", (1,1)) + edge((1,1),(0,2)) + + edge((1,1),(1,2)) + edge((1,1),(2,2))) + g += (node("sagemath-graphs", (0,2)) + node("sagemath-polyhedra", (1,2)) + node("sagemath-singular", (2,2)) + + edge((0,2),(0,3)) + edge((0,2),(1,3)) + edge((1,2),(1,3)) + edge((2,2),(2,3))) + g += (node("sagemath-tdlib", (0,3)) + node("sagemath-standard-no-symbolics", (1,3)) + node("sagemath-symbolics", (2,3)) + + edge((1,3),(1,4)) + edge((2,3),(1,4))) + g += node("sagemath-standard", (1,4)) + sphinx_plot(g, figsize=(8, 4), axes=False) Testing distribution packages From 7d6a44ce9551bfcdde44897c9676281028bafbdb Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Tue, 23 Nov 2021 10:52:15 -0800 Subject: [PATCH 27/30] Use :mod: as markup for packages/modules --- .../en/developer/packaging_sage_library.rst | 70 +++++++++++-------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 2dd5eacedc4..3ba32b30ada 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -11,16 +11,16 @@ Modules, packages, distribution packages The Sage library consists of a large number of Python modules, organized into a hierarchical set of packages that fill the namespace -``sage``. All source files are located in a subdirectory of the +:mod:`sage`. All source files are located in a subdirectory of the directory ``SAGE_ROOT/src/sage/``. For example, - the file ``SAGE_ROOT/src/sage/coding/code_bounds.py`` provides the - module ``sage.coding.code_bounds``; + module :mod:`sage.coding.code_bounds`; - the directory containing this file, ``SAGE_ROOT/src/sage/coding/``, - thus provides the package ``sage.coding``. + thus provides the package :mod:`sage.coding`. There is another notion of "package" in Python, the **distribution package** (also known as a "distribution" or a "pip-installable @@ -38,8 +38,8 @@ mentioned in https://www.python.org/dev/peps/pep-0423/, appears to have largely fallen out of favor, and we will not use it in the SageMath project.) -A distribution that provides Python modules in the ``sage.*`` namespace, say -mainly from ``sage.PAC.KAGE``, should be named **sagemath-DISTRI-BUTION**. +A distribution that provides Python modules in the :mod:`sage.*` namespace, say +mainly from :mod:`sage.PAC.KAGE`, should be named **sagemath-DISTRI-BUTION**. Example: - The distribution https://pypi.org/project/sagemath-categories/ @@ -51,25 +51,25 @@ Other distributions should not use the prefix **sagemath-** in the distribution name. Example: - The distribution https://pypi.org/project/sage-sws2rst/ provides the - Python package ``sage_sws2rst``, so it does not fill the ``sage.*`` + Python package ``sage_sws2rst``, so it does not fill the :mod:`sage.*` namespace. A distribution that provides functionality that does not need to -import anything from the ``sage.*`` namespace should not use the -``sage.*`` namespace for its own packages/modules. It should be +import anything from the :mod:`sage` namespace should not use the +:mod:`sage` namespace for its own packages/modules. It should be positioned as part of the general Python ecosystem instead of as a Sage-specific distribution. Examples: - The distribution https://pypi.org/project/pplpy/ provides the Python - package ``ppl`` and is a much extended version of what used to be - ``sage.libs.ppl``, a part of the Sage library. The package ``sage.libs.ppl`` had + package :mod:`ppl` and is a much extended version of what used to be + :mod:`sage.libs.ppl`, a part of the Sage library. The package :mod:`sage.libs.ppl` had dependencies on :mod:`sage.rings` to convert to/from Sage number types. **pplpy** has no such dependencies and is therefore usable in a wider range of Python projects. - The distribution https://pypi.org/project/memory-allocator/ provides - the Python package ``memory_allocator``. This used to be - ``sage.ext.memory_allocator``, a part of the Sage library. + the Python package :mod:`memory_allocator`. This used to be + :mod:`sage.ext.memory_allocator`, a part of the Sage library. Ordinary packages vs. implicit namespace packages @@ -91,20 +91,20 @@ cannot be an ``__init__.py`` file. For example, -- **sagemath-tdlib** will provide ``sage.graphs.graph_decompositions.tdlib``, +- **sagemath-tdlib** will provide :mod:`sage.graphs.graph_decompositions.tdlib`, -- **sagemath-rw** will provide ``sage.graphs.graph_decompositions.rankwidth``, +- **sagemath-rw** will provide :mod:`sage.graphs.graph_decompositions.rankwidth`, - **sagemath-graphs** will provide all of the rest of - ``sage.graphs.graph_decompositions`` (and most of ``sage.graphs``). + :mod:`sage.graphs.graph_decompositions` (and most of :mod:`sage.graphs`). Then, none of -- ``sage``, +- :mod:`sage`, -- ``sage.graphs``, +- :mod:`sage.graphs`, -- ``sage.graphs.graph_decomposition`` +- :mod:`sage.graphs.graph_decomposition` can be an ordinary package (with an ``__init__.py`` file), but rather each of them has to be an implicit namespace package (no @@ -122,7 +122,7 @@ Source directories of distribution packages =========================================== The development of the Sage library uses a monorepo strategy for -all distribution packages that fill the ``sage.*`` namespace. This +all distribution packages that fill the :mod:`sage.*` namespace. This means that the source trees of these distributions are included in a single ``git`` repository, in a subdirectory of ``SAGE_ROOT/pkgs``. @@ -181,7 +181,7 @@ packages via the file ``pyproject.toml`` ("build-system requires"); this has superseded the older ``setup_requires`` declaration. (There is no mechanism to declare anything regarding the C/C++ libraries.) -While the namespace ``sage.*`` is organized roughly according to +While the namespace :mod:`sage.*` is organized roughly according to mathematical fields or categories, how we partition the implementation modules into distribution packages has to respect the hard constraints that are imposed by the build-time dependencies. @@ -215,17 +215,18 @@ distribution -- which then must be declared as a run-time dependency. *Reducing module-level run-time dependencies:* -- Avoid importing from ``sage.PAC.KAGE.all`` modules when ``sage.PAC.KAGE`` is - a namespace package. The main purpose of the ``*.all`` modules is for +- Avoid importing from :mod:`sage.PAC.KAGE.all` modules when :mod:`sage.PAC.KAGE` is + a namespace package. The main purpose of the :mod:`*.all` modules is for populating the global interactive environment that is available to users at the ``sage:`` prompt. In particular, no Sage library code should import from - ``sage.rings.all``. + :mod:`sage.rings.all`. - Replace module-level imports by method-level imports. Note that this comes with a small runtime overhead, which can become noticeable if the method is called in tight inner loops. -- Sage provides the ``lazy_import`` mechanism. Lazy imports can be +- Sage provides the :func:`~sage.misc.lazy_import.lazy_import` + mechanism. Lazy imports can be declared at the module level, but the actual importing is only done on demand. It is a runtime error at that time if the imported module is not present. This can be convenient compared to local imports in @@ -243,10 +244,17 @@ distribution -- which then must be declared as a run-time dependency. has light dependencies, make ``Class`` a subclass of ``ABC``, and use ``isinstance(object, ABC)``. For example, :mod:`sage.rings.abc` provides abstract base classes for many ring (parent) classes, - including :class:`sage.rings.abc.pAdicField`. So we can replace - ``isinstance(object, sage.rings.padics.generic_nodes.pAdicField)`` - and ``sage.rings.padics.generic_nodes.is_pAdicField(object)`` by - ``isinstance(object, sage.rings.abc.pAdicField)``. + including :class:`sage.rings.abc.pAdicField`. So we can replace:: + + isinstance(object, sage.rings.padics.generic_nodes.pAdicField) + + and:: + + sage.rings.padics.generic_nodes.is_pAdicField(object) + + by:: + + isinstance(object, sage.rings.abc.pAdicField) Other runtime dependencies @@ -276,8 +284,10 @@ around the package :mod:`sage.coding`. First, let's see if it uses symbolics:: Apparently it does not in a very substantial way: -- The imports of the symbolic functions ``ceil`` and ``floor`` can - likely be replaced by :func:`~sage.arith.misc.integer_floor` and +- The imports of the symbolic functions :func:`~sage.functions.other.ceil` + and :func:`~sage.functions.other.floor` can + likely be replaced by the artithmetic functions + :func:`~sage.arith.misc.integer_floor` and :func:`~sage.arith.misc.integer_ceil`. - Looking at the import of ``SR`` by :mod:`sage.coding.grs_code`, it From f83d424d1270d03643d41c47d697c0931b0627dc Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Tue, 23 Nov 2021 10:59:34 -0800 Subject: [PATCH 28/30] Improve ABC example --- src/doc/en/developer/packaging_sage_library.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 3ba32b30ada..6466ea1e599 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -246,14 +246,17 @@ distribution -- which then must be declared as a run-time dependency. provides abstract base classes for many ring (parent) classes, including :class:`sage.rings.abc.pAdicField`. So we can replace:: - isinstance(object, sage.rings.padics.generic_nodes.pAdicField) + from sage.rings.padics.generic_nodes import pAdicField # heavy dependencies + isinstance(object, pAdicField) and:: - sage.rings.padics.generic_nodes.is_pAdicField(object) + from sage.rings.padics.generic_nodes import pAdicField # heavy dependencies + is_pAdicField(object) # deprecated by:: + import sage.rings.abc # no dependencies isinstance(object, sage.rings.abc.pAdicField) From 817a8d0ccc1b65d7bd2d38b2e4b8bcb572ced84f Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Tue, 23 Nov 2021 11:54:49 -0800 Subject: [PATCH 29/30] src/doc/en/developer/packaging_sage_library.rst: More :mod: and :class: markup --- src/doc/en/developer/packaging_sage_library.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index 6466ea1e599..c661b68a841 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -396,16 +396,16 @@ doctest annotation ``# optional``. This mechanism can also be used for making a doctest conditional on the presence of a portion of the Sage library. The available tags take the form of package or module names such as -``sage.combinat``, ``sage.graphs``, ``sage.plot``, ``sage.rings.number_field``, -``sage.rings.real_double``, and ``sage.symbolic``. They are defined via -"features" in a single file, ``SAGE_ROOT/src/sage/features/sagemath.py``, which +:mod:`sage.combinat`, :mod:`sage.graphs`, :mod:`sage.plot`, :mod:`sage.rings.number_field`, +:mod:`sage.rings.real_double`, and :mod:`sage.symbolic`. They are defined via +:class:`~sage.features.Feature` subclasses in the module :mod:`sage.features.sagemath`, which also provides the mapping from features to the distributions providing them (actually, to SPKG names). Using this mapping, Sage can issue installation hints to the user. -For example, the package ``sage.tensor`` is purely algebraic and has +For example, the package :mod:`sage.tensor` is purely algebraic and has no dependency on symbolics. However, there are a small number of -doctests that depend on the Symbolic Ring for integration +doctests that depend on :class:`sage.symbolic.ring.SymbolicRing` for integration testing. Hence, these doctests are marked ``# optional - sage.symbolic``. From b555735c504a86cb0f465afad5daee82db69d77f Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Tue, 23 Nov 2021 16:14:14 -0800 Subject: [PATCH 30/30] src/doc/en/developer/packaging_sage_library.rst: Link to pypi.org and to documentation of packaging metadata --- .../en/developer/packaging_sage_library.rst | 92 ++++++++++++------- 1 file changed, 57 insertions(+), 35 deletions(-) diff --git a/src/doc/en/developer/packaging_sage_library.rst b/src/doc/en/developer/packaging_sage_library.rst index c661b68a841..172e7a9db01 100644 --- a/src/doc/en/developer/packaging_sage_library.rst +++ b/src/doc/en/developer/packaging_sage_library.rst @@ -25,7 +25,8 @@ For example, There is another notion of "package" in Python, the **distribution package** (also known as a "distribution" or a "pip-installable package"). Currently, the entire Sage library is provided by a -single distribution, https://pypi.org/project/sagemath-standard/, +single distribution, +`sagemath-standard `_, which is generated from the directory ``SAGE_ROOT/pkgs/sagemath-standard``. @@ -34,7 +35,7 @@ identifier. In fact, using dashes (``-``) is preferred to underscores in distribution names; **setuptools** and other parts of Python's packaging infrastructure normalize underscores to dashes. (Using dots in distribution names, to indicate ownership by organizations, still -mentioned in https://www.python.org/dev/peps/pep-0423/, appears to +mentioned in `PEP 423 `_, appears to have largely fallen out of favor, and we will not use it in the SageMath project.) @@ -42,17 +43,19 @@ A distribution that provides Python modules in the :mod:`sage.*` namespace, say mainly from :mod:`sage.PAC.KAGE`, should be named **sagemath-DISTRI-BUTION**. Example: -- The distribution https://pypi.org/project/sagemath-categories/ +- The distribution + `sagemath-categories `_ provides a small subset of the modules of the Sage library, mostly - from the packages ``sage.structure``, ``sage.categories``, and - ``sage.misc``. + from the packages :mod:`sage.structure`, :mod:`sage.categories`, and + :mod:`sage.misc`. Other distributions should not use the prefix **sagemath-** in the distribution name. Example: -- The distribution https://pypi.org/project/sage-sws2rst/ provides the - Python package ``sage_sws2rst``, so it does not fill the :mod:`sage.*` - namespace. +- The distribution `sage-sws2rst `_ + provides the Python package :mod:`sage_sws2rst`, so it does not fill + the :mod:`sage.*` namespace and therefore does not use the prefix + **sagemath-**. A distribution that provides functionality that does not need to import anything from the :mod:`sage` namespace should not use the @@ -60,15 +63,15 @@ import anything from the :mod:`sage` namespace should not use the positioned as part of the general Python ecosystem instead of as a Sage-specific distribution. Examples: -- The distribution https://pypi.org/project/pplpy/ provides the Python +- The distribution `pplpy `_ provides the Python package :mod:`ppl` and is a much extended version of what used to be :mod:`sage.libs.ppl`, a part of the Sage library. The package :mod:`sage.libs.ppl` had dependencies on :mod:`sage.rings` to convert to/from Sage number types. **pplpy** has no such dependencies and is therefore usable in a wider range of Python projects. -- The distribution https://pypi.org/project/memory-allocator/ provides - the Python package :mod:`memory_allocator`. This used to be +- The distribution `memory-allocator `_ + provides the Python package :mod:`memory_allocator`. This used to be :mod:`sage.ext.memory_allocator`, a part of the Sage library. @@ -83,7 +86,7 @@ more than one distribution package. By removing the ``__init__.py`` file, however, we can make the package an "implicit" (or "native") "namespace" package, following -https://www.python.org/dev/peps/pep-0420/. Implicit namespace packages can be +`PEP 420 `_. Implicit namespace packages can be included in more than one distribution package. Hence whenever there are two distribution packages that provide modules with a common prefix of Python packages, that prefix needs to be a implicit namespace package, i.e., there @@ -137,24 +140,36 @@ The source directory of a distribution package, such as - ``sage`` -- a relative symbolic link to the monolithic Sage library source tree ``SAGE_ROOT/src/sage/`` -- ``MANIFEST.in`` -- controls which files and directories of the +- `MANIFEST.in `_ -- + controls which files and directories of the monolithic Sage library source tree are included in the distribution -- ``pyproject.toml.m4``, ``setup.cfg.m4``, ``requirements.txt.m4`` -- - Python packaging metadata, declaring the distribution name, dependencies, - etc. (These files are run through the ``m4`` macro processor by the - ``SAGE_ROOT/bootstrap`` script to generate standard files - ``pyproject.toml`` etc.) +- `pyproject.toml `_, + `setup.cfg `_, + and `requirements.txt `_ -- + standard Python packaging metadata, declaring the distribution name, dependencies, + etc. -- ``setup.py`` -- a **setuptools**-based installation script +- ``README.rst`` -- a description of the distribution -- ``tox.ini`` -- testing infrastructure +- ``VERSION.txt``, ``LICENSE.txt`` -- relative symbolic links to the same files + in ``SAGE_ROOT/src`` + +- ``setup.py`` -- a `setuptools `_-based + installation script + +- ``tox.ini`` -- configuration for testing with `tox `_ The technique of using symbolic links pointing into ``SAGE_ROOT/src`` has allowed the modularization effort to keep the ``SAGE_ROOT/src`` tree monolithic: Modularization has been happening behind the scenes and will not change where Sage developers find the source files. +Some of these files may actually be generated from source files with suffix ``.m4`` by the +``SAGE_ROOT/bootstrap`` script via the ``m4`` macro processor. + + + Dependencies and distribution packages ====================================== @@ -177,7 +192,8 @@ must be accessible from the build environment. *Declaring build-time dependencies:* Modern Python packaging provides a mechanism to declare build-time dependencies on other distribution -packages via the file ``pyproject.toml`` ("build-system requires"); this +packages via the file `pyproject.toml `_ +(``[build-system] requires``); this has superseded the older ``setup_requires`` declaration. (There is no mechanism to declare anything regarding the C/C++ libraries.) @@ -211,7 +227,8 @@ modules must be part of the distribution, or provided by another distribution -- which then must be declared as a run-time dependency. *Declaring run-time dependencies:* These dependencies are declared in -``setup.cfg`` (generated from ``setup.cfg.m4``) as ``install_requires``. +``setup.cfg`` (generated from ``setup.cfg.m4``) as +`install_requires `_. *Reducing module-level run-time dependencies:* @@ -318,13 +335,13 @@ features, which will only be working if the user also has installed **sagemath-symbolics**. *Declaring optional run-time dependencies:* It is possible to declare -such optional dependencies as ``extra_requires`` in ``setup.cfg`` +such optional dependencies as `extras_require `_ in ``setup.cfg`` (generated from ``setup.cfg.m4``). This is a very limited mechanism -- in particular it does not affect the build phase of the distribution in any way. It basically only provides a way to give a nickname to a distribution that can be installed as an add-on. -In our example, we could declare an ``extra_requires`` so that users +In our example, we could declare an ``extras_require`` so that users could use ``pip install sagemath-coding[symbolics]``. @@ -343,8 +360,9 @@ FEATURE`` directives in the doctests. Adding these directives will allow developers to test the distribution separately, without requiring all of Sage to be present. -*Declaring doctest-only dependencies:* The ``extra_requires`` mechanism -mentioned above can also be used for this. +*Declaring doctest-only dependencies:* The +`extras_require `_ +mechanism mentioned above can also be used for this. Version constraints of dependencies @@ -352,8 +370,9 @@ Version constraints of dependencies The version information for dependencies comes from the files ``build/pkgs/*/install-requires.txt`` and -``build/pkgs/*/package-version.txt``. We use the ``m4`` macro -processor to insert the version information in the generated files +``build/pkgs/*/package-version.txt``. We use the +`m4 `_ +macro processor to insert the version information in the generated files ``pyproject.toml``, ``setup.cfg``, ``requirements.txt``. @@ -446,7 +465,8 @@ configuration variable settings and the connection to the non-Python packages installed in ``SAGE_LOCAL``. We can now set up a separate virtual environment, in which we install -these wheels and our distribution to be tested. This is where ``tox`` +these wheels and our distribution to be tested. This is where +`tox `_ comes into play: It is the standard Python tool for creating disposable virtual environments for testing. Every distribution in ``SAGE_ROOT/pkgs/`` provides a configuration file ``tox.ini``. @@ -459,7 +479,7 @@ command:: This command does not make any changes to the normal installation of Sage. The virtual environment is created in a subdirectory of -``pkgs/sagemath-standard-no-symbolics/.tox/``. After the command +``SAGE_ROOT/pkgs/sagemath-standard-no-symbolics/.tox/``. After the command finishes, we can start the separate installation of the Sage library in its virtual environment:: @@ -473,25 +493,27 @@ The whole ``.tox`` directory can be safely deleted at any time. We can do the same with other distributions, for example the large distribution **sagemath-standard-no-symbolics** -(https://trac.sagemath.org/ticket/32601), which is intended to provide +(from :trac:`32601`), which is intended to provide everything that is currently in the standard Sage library, i.e., without depending on optional packages, but without the packages -``sage.symbolic``, ``sage.functions``, ``sage.calculus``, etc. +:mod:`sage.symbolic`, :mod:`sage.functions`, :mod:`sage.calculus`, etc. Again we can run the test with ``tox`` in a separate virtual environment:: $ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-standard-no-symbolics && SAGE_NUM_THREADS=16 tox -v -v -v -e py39-sagewheels-nopypi)' Some small distributions, for example the ones providing the two -lowest levels, **sagemath-objects** and **sagemath-categories** -(https://trac.sagemath.org/ticket/29865), can be installed and tested +lowest levels, `sagemath-objects `_ +and `sagemath-categories `_ +(from :trac:`29865`), can be installed and tested without relying on the wheels from the Sage build:: $ ./bootstrap && ./sage -sh -c '(cd pkgs/sagemath-objects && SAGE_NUM_THREADS=16 tox -v -v -v -e py39)' This command finds the declared build-time and run-time dependencies on PyPI, either as source tarballs or as prebuilt wheels, and builds -and installs the distribution **sagemath-objects** in a virtual +and installs the distribution +`sagemath-objects `_ in a virtual environment in a subdirectory of ``pkgs/sagemath-objects/.tox``. Building these small distributions serves as a valuable regression