Developer Guide: Improvements to explanation of version files

src/doc/en/developer/packaging.rst

@@ -579,7 +579,7 @@ and most Python-based packages will also have ``$(PYTHON_TOOLCHAIN)`` as
 an order-only dependency, which will ensure that fundamental packages such
 as ``pip`` and ``setuptools`` are available at the time of building the package.
 
-The best way to install a Python-based package is to use ``pip``, in which
+The best way to install a ``normal`` Python-based package is to use ``pip``, in which
 case the ``spkg-install.in`` script template might just consist of
 
 .. CODE-BLOCK:: bash
@@ -603,16 +603,34 @@ For example, the ``scipy`` ``spkg-check.in`` file contains the line
 
     exec python3 spkg-check.py
 
-All normal Python packages and all wheel packages must have a file ``install-requires.txt``.
-If a Python package is available on PyPI, this file must contain the
-name of the package as it is known to PyPI.  Optionally,
+Abstract requirements: The ``install-requires.txt`` file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+All ``normal`` Python packages and all ``wheel`` packages must have a file
+``install-requires.txt``. For ``pip`` packages, the file is optional; if
+it is missing, the ``requirements.txt`` file is used instead.
+
+If a Python package is available on PyPI, the ``install-requires.txt`` file must
+contain the name of the package as it is known to PyPI.
+
+Optionally,
 ``install-requires.txt`` can encode version constraints (such as lower
 and upper bounds).  The constraints are in the format of the
 ``install_requires`` key of `setup.cfg
 <https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html>`_
 or `setup.py
 <https://packaging.python.org/discussions/install-requires-vs-requirements/#id5>`_.
 
+Sage uses these version constraints for two purposes:
+
+- As a source for generating the metadata of the Python
+  distribution packages in ``SAGE_ROOT/pkgs/``, see
+  :ref:`section_dependencies_distributions`.
+
+- When the experimental option ``configure --enable-system-site-packages`` is used,
+  then the ``configure`` script checks these constraints to determine whether
+  to accept an installation of this package in the system Python.
+
 It is strongly recommended to include comments (starting with ``#``)
 in the file that explain why a particular lower or upper bound is
 warranted or why we wish to include or reject certain versions.
@@ -645,6 +663,87 @@ Setting upper bounds to guard against incompatible future changes is
 a complex topic; see :trac:`33520`.
 
 
+Concrete (pinned) requirements of ``normal``, ``wheel``, ``script`` packages: The ``package-version.txt`` file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Like ``normal`` non-Python packages, all ``normal`` Python packages and all ``wheel`` packages
+must have a file ``package-version.txt``. For ``script`` Python packages, the file is optional.
+
+Sage uses this version for two purposes:
+
+- This is the version that the Sage distribution ships.
+
+- As a source for generating the ``requirements.txt`` files of
+  the Python distribution packages in ``SAGE_ROOT/pkgs/``, see
+  :ref:`section_dependencies_distributions`.
+
+  For the use of the generated ``requirements.txt`` files, see
+  the `pip User Guide <https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_.
+
+
+Concrete requirements of ``pip`` packages: The ``requirements.txt`` file
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In contrast to ``normal``, ``wheel``, and ``script`` packages, the
+``pip`` packages do not use a ``package-version.txt`` file.
+
+Instead, the concrete requirements are set in a ``requirements.txt``
+file, which is passed directly to ``pip`` at installation time.
+
+The ``requirements.txt`` file uses a very flexible format, defined
+in the `pip User Guide
+<https://pip.pypa.io/en/stable/user_guide/#requirements-files>`_.
+Through this format, the concrete requirements can either be
+pinned to a specific version, or set acceptable version ranges, or be
+entirely unconstrained.  The format is even flexible enough to install
+several distribution packages at the same time, and to conditionalize
+on the operating system or Python version.
+
+Pinning a version has the potential benefit of stability, as it can
+avoid retroactive breakage of the Sage distribution by new,
+incompatible versions, and can also help achieve reproducibility
+of computations.
+
+The cost is that updating the version requires
+work by at least two Sage developers: One who prepares a PR and one
+who reviews it.  Moreover, when the package does not get the attention of
+developers who upgrade it, there is the potential risk of missing out
+on bugfixes made in newer versions, or missing out on features in
+major new versions.
+
+Not pinning the version has the obvious potential benefit of always
+being up to date, as ``pip`` contacts the index server (PyPI) to
+obtain and install the package. (Note that ``normal`` and ``wheel``
+packages are always pinned and do not even have access to the index
+server at the time of building and installing the package.)
+
+But this dynamism also brings a risk
+of instability, either by the package itself being affected by bugs in
+a new version, or by breaking compatibility with Sage.
+
+What policy is best for a package depends on various factors,
+including the development velocity and quality control that the
+upstream project uses, the interest by Sage developers in the package,
+the depth of integration in Sage, whether it affects the mathematics,
+etc.
+
+
+Note about dependencies of ``pip`` packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Dependencies of a ``pip`` package do not need to be available as packages
+in the Sage distribution, as the package can pull some of its build-time and
+run-time dependencies directly from PyPI. That's a mild convenience for developers,
+and can be important if one wants to leave the version range wide open.
+
+However, if a dependency is also a package of the Sage distribution,
+then we must declare this dependency.  Otherwise, various errors
+can occur when building or upgrading. When new versions of ``pip``
+packages add dependencies that happen to be Sage packages, there is a
+separate source of instability.
+
+
+
 .. _section-spkg-SPKG-txt:
 
 The SPKG.rst file
@@ -1110,12 +1209,16 @@ For Python packages available from PyPI, you can use::
                                              --type optional
 
 This automatically downloads the most recent version from PyPI and also
-obtains most of the necessary information by querying PyPI.
+obtains most of the necessary information by querying PyPI. In particular,
+the ``SPKG.rst`` file is created as a copy of the package's README file.
+
 
 The ``dependencies`` file may need editing (watch out for warnings regarding
 ``--no-deps`` that Sage issues during installation of the package!).
+
 Also you may want to set lower and upper bounds for acceptable package versions
-in the file ``install-requires.txt``.
+in the file ``install-requires.txt``. (Make sure that the version in
+``package-version.txt`` falls within this acceptable version range!)
 
 By default, when the package is available as a platform-independent
 wheel, the ``sage --package`` creates a wheel package. To create a normal package
@@ -1152,6 +1255,12 @@ For Python packages available from PyPI, there is another shortcut::
     Downloading tarball to ...matplotlib-3.3.1.tar.bz2
     [...............................................................]
 
+When preparing the update, check that any lower and upper bounds for
+acceptable package versions that may be declared in the file
+``install-requires.txt`` are still correct, and update them as needed.
+The version in ``package-version.txt`` always needs to fall within the
+version range!
+
 If you pass the switch ``--commit``, the script will run ``git commit``
 for you.