gh-37500: sage --package, sage-get-system-packages: Support PURLs…

… `pkg:pypi/DISTRO-NAME`, obtain dependencies of wheels from PyPI

    
<!-- ^ Please provide a concise and informative title. -->
<!-- ^ Don't put issue numbers in the title, do this in the PR
description below. -->
<!-- ^ For example, instead of "Fixes #12345" use "Introduce new method
to calculate 1 + 2". -->
<!-- v Describe your changes below in detail. -->
<!-- v Why is this change required? What problem does it solve? -->
<!-- v If this PR resolves an open issue, please link to it here. For
example, "Fixes #12345". -->

We make it possible to refer to Python packages via their PURL (see
[draft PEP 725](https://peps.python.org/pep-0725/#concrete-package-
specification-through-purl)) instead of their SPKG name.

For now a string of the form `pkg:pypi/DISTRO-NAME` is simply a nickname
for the (unique) SPKG that has DISTRO-NAME in their
`version_requirements.txt` or `requirements.txt`. The scheme can also be
omitted: `pypi/DISTRO-NAME` also works. And we also map
`pkg:generic/PACKAGE-NAME` to `PACKAGE_NAME`.

Based on code by @culler, `sage --package create --pypi` now also fills
`dependencies` from the PyPI metadata of wheel packages. When some of
the Python dependencies obtained in this way do not have SPKGs yet, they
are also automatically created.

- Preparation for #31136.
- Split out from #37250.


### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on. For example,
-->
<!-- - #12345: short description why this is a dependency -->
<!-- - #34567: ... -->
    
URL: #37500
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Marc Culler, Matthias Köppe

build/bin/sage-get-system-packages

@@ -50,6 +50,12 @@ case "$SYSTEM" in
         ;;
 esac
 
+case "$SPKGS" in
+    *pkg:*|pypi/*|generic/*)
+        PATH="${SAGE_ROOT}/build/bin:$PATH" SPKGS=$(sage-package list $SPKGS)
+        ;;
+esac
+
 for PKG_BASE in $SPKGS; do
     if [ $FROM_PYPROJECT_TOML -eq 1 ]; then
         if [ -f "$SAGE_ROOT/src/pyproject.toml" ]; then

build/sage_bootstrap/app.py

@@ -10,7 +10,10 @@
 
 
 # ****************************************************************************
-#       Copyright (C) 2016 Volker Braun <vbraun.name@gmail.com>
+#       Copyright (C) 2016      Volker Braun <vbraun.name@gmail.com>
+#                     2020-2024 Matthias Koeppe
+#                     2022      Thierry Monteil
+#                     2024      Marc Culler
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -21,6 +24,7 @@
 
 
 import os
+import re
 import logging
 log = logging.getLogger()
 
@@ -36,6 +40,10 @@
 from sage_bootstrap.env import SAGE_DISTFILES
 
 
+# Approximation of https://peps.python.org/pep-0508/#names dependency specification
+dep_re = re.compile('^ *([-A-Z0-9._]+)', re.IGNORECASE)
+
+
 class Application(object):
 
     def config(self):
@@ -88,7 +96,7 @@ def properties(self, *package_classes, **kwds):
         source_maxima='normal'
         trees_maxima='SAGE_LOCAL'
         """
-        props = kwds.pop('props', ['path', 'version_with_patchlevel', 'type', 'source', 'trees'])
+        props = kwds.pop('props', ['path', 'version_with_patchlevel', 'type', 'source', 'trees', 'purl'])
         format = kwds.pop('format', 'plain')
         log.debug('Looking up properties')
         pc = PackageClass(*package_classes)
@@ -256,6 +264,9 @@ def update_latest(self, package_name, commit=False):
         Update a package to the latest version. This modifies the Sage sources.
         """
         pkg = Package(package_name)
+        if pkg.source not in ['normal', 'wheel']:
+            log.debug('update_latest can only update normal and wheel packages; %s is a %s package' % (pkg, pkg.source))
+            return
         dist_name = pkg.distribution_name
         if dist_name is None:
             log.debug('%s does not have Python distribution info in version_requirements.txt' % pkg)
@@ -380,7 +391,8 @@ def fix_checksum(self, package_name):
             update.fix_checksum()
 
     def create(self, package_name, version=None, tarball=None, pkg_type=None, upstream_url=None,
-               description=None, license=None, upstream_contact=None, pypi=False, source=None):
+               description=None, license=None, upstream_contact=None, pypi=False, source=None,
+               dependencies=None):
         """
         Create a package
 
@@ -392,7 +404,12 @@ def create(self, package_name, version=None, tarball=None, pkg_type=None, upstre
 
         $ sage --package create jupyterlab_markup --pypi --source wheel --type optional
         """
-        if '-' in package_name:
+        if package_name.startswith('pypi/'):
+            package_name = 'pkg:' + package_name
+        if package_name.startswith('pkg:pypi/'):
+            pypi = True
+            package_name = package_name[len('pkg:pypi/'):].lower().replace('-', '_').replace('.', '_')
+        elif '-' in package_name:
             raise ValueError('package names must not contain dashes, use underscore instead')
         if pypi:
             if source is None:
@@ -420,6 +437,24 @@ def create(self, package_name, version=None, tarball=None, pkg_type=None, upstre
                     raise ValueError('Only platform-independent wheels can be used for wheel packages, got {0}'.format(tarball))
                 if not version:
                     version = pypi_version.version
+                if dependencies is None:
+                    requires_dist = pypi_version.requires_dist
+                    if requires_dist:
+                        dependencies = []
+                        for item in requires_dist:
+                            if "extra ==" in item:
+                                continue
+                            try:
+                                dep = dep_re.match(item).groups()[0].strip()
+                            except Exception:
+                                continue
+                            dep = 'pkg:pypi/' + dep
+                            try:
+                                dep = Package(dep).name
+                            except ValueError:
+                                self.create(dep, pkg_type=pkg_type)
+                                dep = Package(dep).name
+                            dependencies.append(dep)
                 upstream_url = 'https://pypi.io/packages/{2}/{0:1.1}/{0}/{1}'.format(package_name, tarball, pypi_version.python_version)
             if not description:
                 description = pypi_version.summary
@@ -444,7 +479,8 @@ def create(self, package_name, version=None, tarball=None, pkg_type=None, upstre
         if description or license or upstream_contact:
             creator.set_description(description, license, upstream_contact)
         if pypi or source == 'pip':
-            creator.set_python_data_and_scripts(pypi_package_name=pypi_version.name, source=source)
+            creator.set_python_data_and_scripts(pypi_package_name=pypi_version.name, source=source,
+                                                dependencies=dependencies)
         if tarball:
             creator.set_tarball(tarball, upstream_url)
             if upstream_url and version:

build/sage_bootstrap/cmdline.py

@@ -12,7 +12,9 @@
 """
 
 # ****************************************************************************
-#       Copyright (C) 2016 Volker Braun <vbraun.name@gmail.com>
+#       Copyright (C) 2015-2016 Volker Braun <vbraun.name@gmail.com>
+#                     2020-2024 Matthias Koeppe
+#                     2022      Thierry Monteil
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -277,9 +279,10 @@ def make_parser():
         formatter_class=argparse.RawDescriptionHelpFormatter,
         help='Print a list of packages known to Sage')
     parser_list.add_argument(
-        'package_class', metavar='[package_name|:package_type:]',
+        'package_class', metavar='[PACKAGE_NAME|pkg:pypi/DISTRIBUTION-NAME|:PACKAGE_TYPE:]',
         type=str, default=[':all-or-nothing:'], nargs='*',
-        help=('package name or designator for all packages of a given type '
+        help=('package name, pkg:pypi/ followed by a distribution name, '
+              'or designator for all packages of a given type '
               '(one of :all:, :standard:, :optional:, and :experimental:); '
               'default: :all: (or nothing when --include-dependencies or --exclude-dependencies is given'))
     parser_list.add_argument(
@@ -305,9 +308,10 @@ def make_parser():
         formatter_class=argparse.RawDescriptionHelpFormatter,
         help='Print properties of given packages')
     parser_properties.add_argument(
-        'package_class', metavar='[package_name|:package_type:]',
+        'package_class', metavar='[PACKAGE_NAME|pkg:pypi/DISTRIBUTION-NAME|:PACKAGE_TYPE:]',
         type=str, nargs='+',
-        help=('package name or designator for all packages of a given type '
+        help=('package name, pkg:pypi/ followed by a distribution name, '
+              'or designator for all packages of a given type '
               '(one of :all:, :standard:, :optional:, and :experimental:)'))
     parser_properties.add_argument(
         '--format', type=str, default='plain',
@@ -410,11 +414,11 @@ def make_parser():
         formatter_class=argparse.RawDescriptionHelpFormatter,
         help='Fix the checksum of normal packages.')
     parser_fix_checksum.add_argument(
-        'package_class', metavar='[package_name|:package_type:]',
+        'package_class', metavar='[PACKAGE_NAME|pkg:pypi/DISTRIBUTION-NAME|:PACKAGE_TYPE:]',
         type=str, default=[':all:'], nargs='*',
-        help=('package name or designator for all packages of a given type '
-              '(one of :all:, :standard:, :optional:, and :experimental:); '
-              'default: :all:'))
+        help=('package name, pkg:pypi/ followed by a distribution name, '
+              'or designator for all packages of a given type '
+              '(one of :all:, :standard:, :optional:, and :experimental:; default: :all:)'))
 
     parser_create = subparsers.add_parser(
         'create', epilog=epilog_create,
@@ -453,9 +457,10 @@ def make_parser():
         formatter_class=argparse.RawDescriptionHelpFormatter,
         help='Print metrics of given packages')
     parser_metrics.add_argument(
-        'package_class', metavar='[package_name|:package_type:]',
+        'package_class', metavar='[PACKAGE_NAME|pkg:pypi/DISTRIBUTION-NAME|:PACKAGE_TYPE:]',
         type=str, nargs='*', default=[':all:'],
-        help=('package name or designator for all packages of a given type '
+        help=('package name, pkg:pypi/ followed by a distribution name, '
+              'or designator for all packages of a given type '
               '(one of :all:, :standard:, :optional:, and :experimental:; default: :all:)'))
 
     return parser

build/sage_bootstrap/creator.py

@@ -4,7 +4,8 @@
 """
 
 # ****************************************************************************
-#       Copyright (C) 2016 Volker Braun <vbraun.name@gmail.com>
+#       Copyright (C) 2015-2016 Volker Braun <vbraun.name@gmail.com>
+#                     2020-2024 Matthias Koeppe
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -105,7 +106,7 @@ def _remove_files(self, files):
             except OSError:
                 pass
 
-    def set_python_data_and_scripts(self, pypi_package_name=None, source='normal'):
+    def set_python_data_and_scripts(self, pypi_package_name=None, source='normal', dependencies=None):
         """
         Write the file ``dependencies`` and other files for Python packages.
 
@@ -121,7 +122,15 @@ def set_python_data_and_scripts(self, pypi_package_name=None, source='normal'):
         if pypi_package_name is None:
             pypi_package_name = self.package_name
         with open(os.path.join(self.path, 'dependencies'), 'w+') as f:
-            f.write(' | $(PYTHON_TOOLCHAIN) $(PYTHON)\n\n')
+            if dependencies:
+                dependencies = ' '.join(dependencies)
+            else:
+                dependencies = ''
+            if source == 'wheel':
+                dependencies_order_only = 'pip $(PYTHON)'
+            else:
+                dependencies_order_only = '$(PYTHON_TOOLCHAIN) $(PYTHON)'
+            f.write(dependencies + ' | ' + dependencies_order_only + '\n\n')
             f.write('----------\nAll lines of this file are ignored except the first.\n')
         if source == 'normal':
             with open(os.path.join(self.path, 'spkg-install.in'), 'w+') as f:

build/sage_bootstrap/download/cmdline.py

@@ -6,7 +6,9 @@
 """
 
 # ****************************************************************************
-#       Copyright (C) 2016 Volker Braun <vbraun.name@gmail.com>
+#       Copyright (C) 2015-2016 Volker Braun <vbraun.name@gmail.com>
+#                     2015      Jeroen Demeyer
+#                     2020      Matthias Koeppe
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

build/sage_bootstrap/download/mirror_list.py

@@ -4,7 +4,9 @@
 """
 
 #*****************************************************************************
-#       Copyright (C) 2015 Volker Braun <vbraun.name@gmail.com>
+#       Copyright (C) 2014-2016 Volker Braun <vbraun.name@gmail.com>
+#                     2015      Jeroen Demeyer
+#                     2023      Matthias Koeppe
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

build/sage_bootstrap/expand_class.py

@@ -4,7 +4,8 @@
 """
 
 # ****************************************************************************
-#       Copyright (C) 2016 Volker Braun <vbraun.name@gmail.com>
+#       Copyright (C) 2016      Volker Braun <vbraun.name@gmail.com>
+#                     2020-2024 Matthias Koeppe
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -52,12 +53,21 @@ def included_in_filter(pkg):
                 self._init_optional(predicate=included_in_filter)
             elif package_name_or_class == ':experimental:':
                 self._init_experimental(predicate=included_in_filter)
+            elif any(package_name_or_class.startswith(prefix)
+                     for prefix in ["pkg:", "pypi/", "generic"]):
+                self.__names.add(Package(package_name_or_class).name)
             else:
                 if ':' in package_name_or_class:
-                    raise ValueError('a colon may only appear in designators of package types, '
+                    raise ValueError('a colon may only appear in a PURL such as '
+                                     'pkg:pypi/DISTRIBUTION-NAME '
+                                     'and in designators of package types, '
                                      'which must be one of '
                                      ':all:, :standard:, :optional:, or :experimental:'
                                      'got {}'.format(package_name_or_class))
+                if '-' in package_name_or_class:
+                    raise ValueError('dashes may only appear in a PURL such as '
+                                     'pkg:pypi/DISTRIBUTION-NAME; '
+                                     'SPKG names use underscores')
                 self.__names.add(package_name_or_class)
 
         def include_recursive_dependencies(names, package_name):