diff --git a/peps/pep-0725.rst b/peps/pep-0725.rst index 53a11b86c02..2a6160433ea 100644 --- a/peps/pep-0725.rst +++ b/peps/pep-0725.rst @@ -1,6 +1,7 @@ PEP: 725 Title: Specifying external dependencies in pyproject.toml Author: Pradyun Gedam , + Jaime Rodríguez-Guerra , Ralf Gommers Discussions-To: https://discuss.python.org/t/31888 Status: Draft @@ -18,15 +19,21 @@ runtime dependencies in a ``pyproject.toml`` file for packaging-related tools to consume. This PEP proposes to add an ``[external]`` table to ``pyproject.toml`` with -three keys: "build-requires", "host-requires" and "dependencies". These -are for specifying three types of dependencies: +seven keys. "build-requires", "host-requires" and "dependencies" +are for specifying three types of *required* dependencies: 1. ``build-requires``, build tools to run on the build machine -2. ``host-requires``, build dependencies needed for host machine but also needed at build time. +2. ``host-requires``, build dependencies needed for the host machine but also needed at build time. 3. ``dependencies``, needed at runtime on the host machine but not needed at build time. -Cross compilation is taken into account by distinguishing build and host dependencies. -Optional build-time and runtime dependencies are supported too, in a manner analogies +These three keys also have their *optional* ``external`` counterparts (``optional-build-requires``, +``optional-host-requires``, ``optional-dependencies``), which have the same role that +``project.optional-dependencies`` plays for ``project.dependencies``. Finally, +``dependency-groups`` offers the same functionality as :pep:`735` but for external +dependencies. + +Cross compilation is taken into account by distinguishing between build and host dependencies. +Optional build-time and runtime dependencies are supported too, in a manner analogous to how that is supported in the ``[project]`` table. @@ -41,25 +48,30 @@ this PEP are to: - Enable tools to automatically map external dependencies to packages in other packaging repositories, -- Make it possible to include needed dependencies in error messages emitting by +- Make it possible to include needed dependencies in error messages emitted by Python package installers and build frontends, - Provide a canonical place for package authors to record this dependency information. -Packaging ecosystems like Linux distros, Conda, Homebrew, Spack, and Nix need +Packaging ecosystems like Linux distros, conda, Homebrew, Spack, and Nix need full sets of dependencies for Python packages, and have tools like pyp2spec_ -(Fedora), Grayskull_ (Conda), and dh_python_ (Debian) which attempt to +(Fedora), Grayskull_ (conda), and dh_python_ (Debian) which attempt to automatically generate dependency metadata for their own package managers from the metadata in upstream Python packages. External dependencies are currently handled manually, because there is no metadata for this in ``pyproject.toml`` or any other -standard location. Enabling automating this conversion is a key benefit of -this PEP, making packaging Python packages for distros easier and more reliable. In addition, the -authors envision other types of tools making use of this information, e.g., -dependency analysis tools like Repology_, Dependabot_ and libraries.io_. +standard location. Other tools resort to extracting dependencies from extension +modules and shared libraries inside Python packages, like elfdeps_ (Fedora). +Enabling automating this type of conversion by only using explicitly annotated metadata +is a key benefit of this PEP, making packaging Python packages for distros easier +and more reliable. In addition, the authors envision other types of tools +making use of this information, e.g., dependency analysis tools like Repology_, +Dependabot_ and libraries.io_. + Software bill of materials (SBOM) generation tools may also be able to use this information, e.g. for flagging that external dependencies listed in ``pyproject.toml`` but not contained in wheel metadata are likely vendored -within the wheel. +within the wheel. :pep:`770`, which standardizes how SBOMs are included in +wheels, contains an instructive section on how that PEP differs from this one. Packages with external dependencies are typically hard to build from source, and error messages from build failures tend to be hard to decipher for end @@ -76,7 +88,8 @@ information will improve this situation. This PEP is not trying to specify how the external dependencies should be used, nor a mechanism to implement a name mapping from names of individual packages that are canonical for Python projects published on PyPI to those of other -packaging ecosystems. Those topics should be addressed in separate PEPs. +packaging ecosystems. Canonical names and a name mapping mechanism are addressed +in :pep:`804`. Rationale @@ -101,14 +114,14 @@ Multiple types of external dependencies can be distinguished: concrete packages. E.g., a C++ compiler, BLAS, LAPACK, OpenMP, MPI. Concrete packages are straightforward to understand, and are a concept present -in virtually every package management system. Virtual packages are a concept +in every package management system. Virtual packages are a concept also present in a number of packaging systems -- but not always, and the -details of their implementation varies. +details of their implementation vary. Cross compilation ----------------- -Cross compilation is not yet (as of August 2023) well-supported by stdlib +Cross compilation is not yet (as of September 2025) well-supported by stdlib modules and ``pyproject.toml`` metadata. It is however important when translating external dependencies to those of other packaging systems (with tools like ``pyp2spec``). Introducing support for cross compilation immediately @@ -121,23 +134,27 @@ Terminology This PEP uses the following terminology: - *build machine*: the machine on which the package build process is being - executed + executed. - *host machine*: the machine on which the produced artifact will be installed - and run -- *build dependency*: dependency for building the package that needs to be - present at build time and itself was built for the build machine's OS and - architecture -- *host dependency*: dependency for building the package that needs to be - present at build time and itself was built for the host machine's OS and - architecture + and run. +- *build dependency*: package required only during the build process. It must + be available at build time and is built for the *build* machine's OS and + architecture. Typical examples include compilers, code generators, and + build tools. +- *host dependency*: package needed during the build and often also at runtime. + It must be available during the build and is built for the *host* machine's OS + and architecture. These are usually libraries the project links against. +- *runtime dependency*: package required only when the package is used after + installation. It is not required at build time but must be available on + the *host* machine at runtime. Note that this terminology is not consistent across build and packaging tools, so care must be taken when comparing build/host dependencies in ``pyproject.toml`` to dependencies from other package managers. -Note that "target machine" or "target dependency" is not used in this PEP. That -is typically only relevant for cross-compiling compilers or other such advanced -scenarios [#gcc-cross-terminology]_, [#meson-cross]_ - this is out of scope for +Note that "target machine" or "target dependency" are not used in this PEP. That +is typically only relevant for cross-compiling a compiler or other such advanced +scenarios [#gcc-cross-terminology]_, [#meson-cross]_ -- this is out of scope for this PEP. Finally, note that while "dependency" is the term most widely used for packages @@ -148,8 +165,8 @@ build-time dependencies is ``build-requires``. Hence this PEP uses the keys Build and host dependencies ''''''''''''''''''''''''''' -Clear separation of metadata associated with the definition of build and target -platforms, rather than assuming that build and target platform will always be +Clear separation of metadata associated with the definition of build and host +platforms, rather than assuming that build and host platform will always be the same, is important [#pypackaging-native-cross]_. Build dependencies are typically run during the build process - they may be @@ -169,130 +186,168 @@ necessary to run a host dependency under an emulator, or through a custom tool like crossenv_. When host dependencies imply a runtime dependency, that runtime dependency also does not have to be declared, just like for build dependencies. -When host dependencies are declared and a tool is not cross-compilation aware -and has to do something with external dependencies, the tool MAY merge the -``host-requires`` list into ``build-requires``. This may for example happen if -an installer like ``pip`` starts reporting external dependencies as a likely -cause of a build failure when a package fails to build from an sdist. +When host dependencies are declared and a tool which is executing an action +unrelated to cross-compiling, it may decide to merge the ``host-requires`` list +into ``build-requires`` - whether this is useful is context-dependent. Specifying external dependencies -------------------------------- -Concrete package specification through PURL -''''''''''''''''''''''''''''''''''''''''''' +Concrete package specification +'''''''''''''''''''''''''''''' -The two types of concrete packages are supported by PURL_ (Package URL), which -implements a scheme for identifying packages that is meant to be portable -across packaging ecosystems. Its design is:: +A "Package URL" or `PURL`_ is a widely used URL string for identifying packages +that is meant to be portable across packaging ecosystems. Its design is:: scheme:type/namespace/name@version?qualifiers#subpath The ``scheme`` component is a fixed string, ``pkg``, and of the other -components only ``type`` and ``name`` are required. As an example, a package -URL for the ``requests`` package on PyPI would be:: +components only ``type`` and ``name`` are required. + +Since external dependencies are likely to be typed by hand, we propose a PURL +derivative that, in the name of ergonomics and user-friendliness, introduces a +number of changes (further discussed below): + +- Support for virtual packages via a new ``virtual`` type. +- Allow version ranges (and not just literals) in the ``version`` field. - pkg:pypi/requests +In this derivative, we replace the ``pkg`` scheme with ``dep``. Hence, +we will refer to them as DepURLs. -Adopting PURL to specify external dependencies in ``pyproject.toml`` solves a -number of problems at once - and there are already implementations of the -specification in Python and multiple languages. PURL is also already supported -by dependency-related tooling like SPDX (see +As an example, a DepURL for the ``requests`` package on PyPI would be:: + + dep:pypi/requests + # equivalent to pkg:pypi/requests + +Adopting PURL-compatible strings to specify external dependencies in +``pyproject.toml`` solves a number of problems at once, and there are already +implementations of the specification in Python and multiple other languages. PURL is +also already supported by dependency-related tooling like SPDX (see `External Repository Identifiers in the SPDX 2.3 spec `__), the `Open Source Vulnerability format `__, and the `Sonatype OSS Index `__; not having to wait years before support in such tooling arrives is valuable. +DepURLs are very easily transformed into PURLs, with the exception of +``dep:virtual`` which doesn't have an equivalent in `PURL`_. For concrete packages without a canonical package manager to refer to, either -``pkg:generic/pkg-name`` can be used, or a direct reference to the VCS system +``dep:generic/dep-name`` can be used, or a direct reference to the VCS system that the package is maintained in (e.g., -``pkg:github/user-or-org-name/pkg-name``). Which of these is more appropriate -is situation-dependent. This PEP recommends using ``pkg:generic`` when the -package name is unambiguous and well-known (e.g., ``pkg:generic/git`` or -``pkg:generic/openblas``), and using the VCS as the PURL type otherwise. +``dep:github/user-or-org-name/dep-name``). Which of these is more appropriate +is situation-dependent. This PEP recommends using ``dep:generic`` when the +package name is unambiguous and well-known (e.g., ``dep:generic/git`` or +``dep:generic/openblas``), and using the VCS as the type otherwise. Which name +is chosen as canonical for any given package, as well as the process to make +and record such choices, is the topic of :pep:`804`. Virtual package specification -''''''''''''''''''''''''''''' +'''''''''''''''''''''''''''''' + +PURL does not offer support for virtual or virtual dependency specification yet. +A `proposal to add a virtual type `__ +is being discussed for revision 1.1. + +In the meantime, we propose adding a new *type* to our ``dep:`` derivative, the ``virtual`` +type, which can take two *namespaces* (extensible through the process given in +:pep:`804`): -There is no ready-made support for virtual packages in PURL or another -standard. There are a relatively limited number of such dependencies though, -and adopting a scheme similar to PURL but with the ``virtual:`` rather than -``pkg:`` scheme seems like it will be understandable and map well to Linux -distros with virtual packages and to the likes of Conda and Spack. +- ``interface``: for components such as BLAS or MPI. +- ``compiler``: for compiled languages like C or Rust. -The two known virtual package types are ``compiler`` and ``interface``. +The *name* should be the most common name for the interface or language, lowercased. +Some examples include:: + + dep:virtual/compiler/c + dep:virtual/compiler/cxx + dep:virtual/compiler/rust + dep:virtual/interface/blas + dep:virtual/interface/lapack + +Since there are a limited number of such dependencies, it seems like it will be +understandable and map well to Linux distros with virtual packages and to the +likes of conda and Spack. Versioning '''''''''' +PURLs support fixed versions via the ``@`` component of the URL. For example, +``numpy===2.0`` can be expressed as ``pkg:pypi/numpy@2.0``. + Support in PURL for version expressions and ranges beyond a fixed version is -still pending, see the Open Issues section. +available via ``vers`` URIs (`see specification `__):: -Dependency specifiers -''''''''''''''''''''' + vers:type/version-constraint|version-constraint|... -Regular Python dependency specifiers (as originally defined in :pep:`508`) may -be used behind PURLs. PURL qualifiers, which use ``?`` followed by a package -type-specific dependency specifier component, must not be used. The reason for -this is pragmatic: dependency specifiers are already used for other metadata in -``pyproject.toml``, any tooling that is used with ``pyproject.toml`` is likely -to already have a robust implementation to parse it. And we do not expect to -need the extra possibilities that PURL qualifiers provide (e.g. to specify a -Conan or Conda channel, or a RubyGems platform). +Users are supposed to couple a ``pkg:`` URL with a ``vers:`` URL. For example, +to express ``numpy>=2.0``, the PURL equivalent would be ``pkg:pypi/numpy`` plus +``vers:pypi/>=2.0``. This can be done with: -Usage of core metadata fields ------------------------------ +- A two-item list: ``["pkg:pypi/numpy", "vers:pypi/>=2.0"]``. +- A `percent-encoded `__ + URL qualifier: ``pkg:pypi/numpy?vers=vers:pypi%2F%3E%3D2.0``. -The `core metadata`_ specification contains one relevant field, namely -``Requires-External``. This has no well-defined semantics in core metadata 2.1; -this PEP chooses to reuse the field for external runtime dependencies. The core -metadata specification does not contain fields for any metadata in -``pyproject.toml``'s ``[build-system]`` table. Therefore the ``build-requires`` -and ``host-requires`` content also does not need to be reflected in core -metadata fields. The ``optional-dependencies`` content from ``[external]`` -would need to either reuse ``Provides-Extra`` or require a new -``Provides-External-Extra`` field. Neither seems desirable. - -Differences between sdist and wheel metadata -'''''''''''''''''''''''''''''''''''''''''''' +Since none of these options are very ergonomic, we chose instead for DepURLs +to accept version range specifiers too with semantics that are a subset of +:pep:`440` semantics. The allowed operators are those that are widely available +across package managers (e.g., ``==``, ``>`` and ``>=`` are common, while +``~=`` isn't). -A wheel may vendor its external dependencies. This happens in particular when -distributing wheels on PyPI or other Python package indexes - and tools like -auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a -``Requires-External`` entry in an sdist may disappear from a wheel built from -that sdist. It is also possible that a ``Requires-External`` entry remains in a -wheel, either unchanged or with narrower constraints. ``auditwheel`` does not -vendor certain allow-listed dependencies, such as OpenGL, by default. In -addition, ``auditwheel`` and ``delvewheel`` allow a user to manually exclude -dependencies via a ``--exclude`` or ``--no-dll`` command-line flag. This is -used to avoid vendoring large shared libraries, for example those from CUDA. - -``Requires-External`` entries generated from external dependencies in -``pyproject.toml`` in a wheel are therefore allowed to be narrower than those -for the corresponding sdist. They must not be wider, i.e. constraints must not -allow a version of a dependency for a wheel that isn't allowed for an sdist, -nor contain new dependencies that are not listed in the sdist's metadata at -all. +Some examples: + +- ``dep:pypi/numpy@2.0``: ``numpy`` pinned at exactly version 2.0. +- ``dep:pypi/numpy@>=2.0``: ``numpy`` with version greater or equal than 2.0. +- ``dep:virtual/interface/lapack@>=3.7.1``: any package implementing the + LAPACK interface for version greater or equal than ``3.7.1``. + +The versioning scheme for particular virtual packages, in case that isn't +unambiguously defined by an upstream project or standard, will be defined in +the Central Registry (see :pep:`804`). + +Environment markers +''''''''''''''''''' + +Regular environment markers (as originally defined in :pep:`508`) may +be used behind DepURLs. PURL qualifiers, which use ``?`` followed by a package +type-specific dependency specifier component, should not be used for the +purposes for which environment markers suffice. The reason for this is +pragmatic: environment markers are already used for other metadata in +``pyproject.toml``, hence any tooling that is used with ``pyproject.toml`` is +likely to already have a robust implementation to parse it. And we do not +expect to need the extra possibilities that PURL qualifiers provide (e.g., to +specify a Conan or conda channel, or a RubyGems platform). + +We name the combination of a DepURL and environment markers as "external +dependency specifiers", analogously to the existing `dependency specifiers`_. Canonical names of dependencies and ``-dev(el)`` split packages ''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -It is fairly common for distros to split a package into two or more packages. -In particular, runtime components are often separately installable from -development components (headers, pkg-config and CMake files, etc.). The latter -then typically has a name with ``-dev`` or ``-devel`` appended to the -project/library name. This split is the responsibility of each distro to -maintain, and should not be reflected in the ``[external]`` table. It is not -possible to specify this in a reasonable way that works across distros, hence -only the canonical name should be used in ``[external]``. - -The intended meaning of using a PURL or virtual dependency is "the full package -with the name specified". It will depend on the context in which the metadata -is used whether the split is relevant. For example, if ``libffi`` is a host +It is fairly common, but far from universal, for distros to split a package +into two or more packages. In particular, runtime components are often +separately installable from development components (headers, pkg-config and +CMake files, etc.). The latter then typically has a name with ``-dev`` or +``-devel`` appended to the project/library name. Also, larger packages are +sometimes split into multiple separate packages to keep install sizes +manageable. More often than not, such package splits are not defined or +recognized by the maintainers of a package, and it's therefore ambiguous what +any split would mean. Hence, such splits should not be reflected in the +``[external]`` table. It is not possible to specify this in a reasonable way +that works across distros, hence only the canonical name should be used in +``[external]``. + +The intended meaning of using a DepURL is "the full package with the name +specified". I.e., including all installable artifacts that are part of the +package. It will depend on the context in which the metadata is used whether a +package split is relevant. For example, if ``libffi`` is a host dependency and a tool wants to prepare an environment for building a wheel, then if a distro has split off the headers for ``libffi`` into a ``libffi-devel`` package then the tool has to install both ``libffi`` and ``libffi-devel``. +For defining what canonical package names are and how package splits are +handled in practice when tools attempt to use ``[external]`` for installation +purposes, we refer to :pep:`804`. + Python development headers '''''''''''''''''''''''''' @@ -306,6 +361,98 @@ consistency between Python dependencies and external dependencies, we choose to add it implicitly. Python development headers must be assumed to be necessary when an ``[external]`` table contains one or more compiler packages. +New Core Metadata fields +------------------------ + +Two new Core Metadata fields are proposed: + +- ``Requires-External-Dep``. An external requirement. Mimics the transition + from ``Requires`` to ``Requires-Dist``. We chose the ``-Dep`` suffix to + emphasize that the value is not a regular Python specifier (distribution), + but an external dependency specifier containing a DepURL. +- ``Provides-External-Extra``. An *extra* group that carries external dependencies + (as found in ``Requires-External-Dep``) only. + +Since the Core Metadata specification does not contain fields for any metadata in +``pyproject.toml``'s ``[build-system]`` table, the ``build-requires`` +and ``host-requires`` content do not need to be reflected in existing core +metadata fields. + +Additionally, this PEP also proposes to deprecate the ``Requires-External`` field. +The reasons being: + +- Avoiding confusion with the newly proposed fields. +- Avoiding potential incompatibilities with existing usage (even if limited). +- Low penetration in the ecosystem: + + - There is no direct correspondence to a field in the ``pyproject.toml`` metadata. + - Mainstream build backends like ``setuptools`` (see `pypa/setuptools#4220`_), + ``hatch`` (see `pypa/hatch#1712`_), ``flit`` (see `pypa/flit#353`_), or ``poetry`` + do not offer ways to specify it or require a plugin (e.g. `poetry-external-dependencies`_). + ``maturin`` does seem to support it since 0.7.0 (see `PyO3/maturin@5b0e4808`_), + but it's not directly documented. Other backends like ``scikit-build-core`` or + ``meson-python`` returned no results for ``External-Requires``. + - The field is not included in the `PyPI JSON API responses`_. + +Effect of vendoring shared libraries on wheel metadata +'''''''''''''''''''''''''''''''''''''''''''''''''''''' + +A wheel may vendor its external dependencies. This happens in particular when +distributing wheels on PyPI or other Python package indexes -- and tools like +auditwheel_, delvewheel_ and delocate_ automate this process. As a result, a +``Requires-External-Dep`` entry in an sdist may disappear from a wheel built from +that sdist with a tool like ``cibuildwheel``. It is also possible that a +``Requires-External-Dep`` entry remains in a wheel, either unchanged or with +narrower constraints. ``auditwheel`` does not vendor certain allow-listed +dependencies, such as OpenGL, by default. In addition, ``auditwheel`` and +``delvewheel`` allow a user to manually exclude dependencies via a +``--exclude`` or ``--no-dll`` command-line flag. This is used to avoid +vendoring large shared libraries, for example those from CUDA. + +``Requires-External-Dep`` entries generated from external dependencies in +``pyproject.toml`` can therefore differ between an sdist and its corresponding +wheel(s) depending on the build/distribution process. + +Note that this does not imply that the field must be marked as Dynamic, since +this distinction only applies to wheels built from an sdist by a build backend. +In particular, wheels built from other wheels do not need to satisfy this +constraint. + +Dependency groups +----------------- + +This PEP has chosen to include the :pep:`735` key ``dependency-groups`` under +the ``[external]`` table too. This decision is motivated by the need of having +similar functionality for external metadata. The top-level table cannot be used +for external dependencies because it's expected to have PEP 508 strings (and tables +for group includes), while we have chosen to rely on ``dep:`` URLs for the external +dependencies. Conflating both would raise significant backwards compatibility +issues with existing usage. + +Strictly speaking, the ``dependency-groups`` schema allows us to define external +dependencies in per-group sub-tables:: + + [dependency-groups] + dev = [ + "pytest", + { external = ["dep:cargo/ripgrep"] }, + ] + +However, this has the same problem: we are mixing different types of dependency +specifiers in the same data structure. We believe it's cleaner to separate concerns +in different top-level tables, hence why we still prefer to have +``external.dependency-groups``. + +Optional dependencies versus dependency groups +'''''''''''''''''''''''''''''''''''''''''''''' + +The rationale for having ``external.dependency-groups`` is identical for the +rationale given in :pep:`735` for introducing ``[dependency-groups]``. The +intended usage and semantics of inclusion/exclusion into Core Metadata +is thus identical to ``[dependency-groups]``. + +``external.optional-dependencies`` will show up in Core Metadata. +``external.dependency-groups`` will not. Specification ============= @@ -313,8 +460,70 @@ Specification If metadata is improperly specified then tools MUST raise an error to notify the user about their mistake. -Details -------- +DepURL +------ + +A DepURL implements a scheme for identifying packages that is meant to be +portable across packaging ecosystems. Its design is:: + + dep:type/namespace/name@version?qualifiers#subpath + +``dep:`` is a fixed string, and always present. ``type`` and ``name`` are +required, other components are optional. All components apply for both PURL +and virtual ``type``'s, and have these requirements: + +- ``type`` (required): MUST be either a `PURL`_ ``type``, or ``virtual``. +- ``namespace`` (optional): MUST be a `PURL`_ ``namespace``, or a namespace in + the DepURL central registry (see :pep:`804`). +- ``name`` (required): MUST be a name that parses as a valid `PURL`_ ``name``. + Tools MAY warn or error if a name is not present in the DepURL central + registry (see :pep:`804`). +- ``version`` (optional): MUST be a regular `version specifier`_ (PEP 440 + semantics) as a single version or version range, with the restriction that + only the following operators may be used: ``>=``, ``>``, ``<``, ``<=``, + ``==``, ``,``. +- ``qualifiers`` (optional): MUST parse as a valid `PURL`_ ``qualifier``. +- ``subpath`` (optional): MUST parse as a valid `PURL`_ ``subpath``. + +External dependency specifiers +------------------------------ + +External dependency specifiers MUST contain a DepURL, and MAY contain +environment markers with the same syntax as used in regular `dependency +specifiers`_ (as originally specified in :pep:`508`). + + +Changes in Core Metadata +------------------------ + +Deprecations +'''''''''''' + +The ``External-Requires`` Core Metadata field will be marked as *obsolete* and its +usage will be discouraged. + +Additions +''''''''' + +Two new fields are added to Core Metadata: + +- ``Requires-External-Dep``. An external requirement expressed as an external + dependency specifier string. +- ``Provides-External-Extra``. An *extra* group that carries external dependencies + (as found in ``Requires-External-Dep``) only. + +Version bump +'''''''''''' + +Given that the proposed changes are purely additive, the Core Metadata +version will be bumped to 2.6. + +This will only impact PyPI and tools that want to support external runtime dependencies, +and require no changes otherwise. + + +Changes in ``pyproject.toml`` +----------------------------- Note that ``pyproject.toml`` content is in the same format as in :pep:`621`. @@ -330,121 +539,159 @@ to be present on the system already. ``build-requires``/``optional-build-requires`` '''''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``build-requires``) and a table - with values of arrays of PURL_ strings (``optional-build-requires``) +- Format: Array of external dependency specifiers (``build-requires``) and a + table with values of arrays of external dependency specifiers + (``optional-build-requires``) - `Core metadata`_: N/A The (optional) external build requirements needed to build the project. For ``build-requires``, it is a key whose value is an array of strings. Each string represents a build requirement of the project and MUST be formatted as -either a valid PURL_ string or a ``virtual:`` string. +a valid external dependency specifier. For ``optional-build-requires``, it is a table where each key specifies an extra set of build requirements and whose value is an array of strings. The -strings of the arrays MUST be valid PURL_ strings. +strings of the arrays MUST be valid external dependency specifiers. ``host-requires``/``optional-host-requires`` '''''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``host-requires``) and a table - with values of arrays of PURL_ strings (``optional-host-requires``) -- `Core metadata`_: N/A +- Format: Array of external dependency specifiers (``host-requires``) and a + table with values of arrays of external dependency specifiers + (``optional-host-requires``) - + `Core metadata`_: N/A The (optional) external host requirements needed to build the project. For ``host-requires``, it is a key whose value is an array of strings. Each string represents a host requirement of the project and MUST be formatted as -either a valid PURL_ string or a ``virtual:`` string. +a valid external dependency specifier. For ``optional-host-requires``, it is a table where each key specifies an extra set of host requirements and whose value is an array of strings. The -strings of the arrays MUST be valid PURL_ strings. +strings of the arrays MUST be valid external dependency specifiers. ``dependencies``/``optional-dependencies`` '''''''''''''''''''''''''''''''''''''''''' -- Format: Array of PURL_ strings (``dependencies``) and a table - with values of arrays of PURL_ strings (``optional-dependencies``) -- `Core metadata`_: ``Requires-External``, N/A +- Format: Array of external dependency specifiers (``dependencies``) and a + table with values of arrays of external dependency specifiers + (``optional-dependencies``) +- `Core metadata`_: ``Requires-External-Dep``, ``Provides-External-Extra`` The (optional) runtime dependencies of the project. For ``dependencies``, it is a key whose value is an array of strings. Each -string represents a dependency of the project and MUST be formatted as either a -valid PURL_ string or a ``virtual:`` string. Each string maps directly to a -``Requires-External`` entry in the `core metadata`_. +string represents a dependency of the project and MUST be formatted as a valid +external dependency specifier. Each string must be added to `Core Metadata`_ as +a ``Requires-External-Dep`` field. -For ``optional-dependencies``, it is a table where each key specifies an extra +For ``optional-dependencies``, it is a table where each key specifies an *extra* and whose value is an array of strings. The strings of the arrays MUST be valid -PURL_ strings. Optional dependencies do not map to a core metadata field. +external dependency specifiers. For each ``optional-dependencies`` group: + +- The name of the group MUST be added to `Core Metadata`_ as a + ``Provides-External-Extra`` field. +- The external dependency specifiers in that group MUST be added to `Core + Metadata`_ as a ``Requires-External-Dep`` field, with the corresponding ``; + extra == 'name'`` environment marker. + +``dependency-groups`` +''''''''''''''''''''' + +- Format: A table where each key is the name of the group, and the values are + arrays of external dependency specifiers, tables, or a mix of both. +- `Core metadata`_: N/A + +PEP 735 -style dependency groups, but using external dependency specifiers +instead of PEP 508 strings. Every other detail (e.g. group inclusion, name +normalization) follows the official `dependency groups specification`_. Examples -------- -These examples show what the ``[external]`` content for a number of packages is +These examples show what the ``[external]`` table content for a number of +packages, and the corresponding ``PKG-INFO``/``METADATA`` content (if any) is expected to be. -cryptography 39.0: +cryptography 39.0 +''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] build-requires = [ - "virtual:compiler/c", - "virtual:compiler/rust", - "pkg:generic/pkg-config", + "dep:virtual/compiler/c", + "dep:virtual/compiler/rust", + "dep:generic/pkg-config", ] host-requires = [ - "pkg:generic/openssl", - "pkg:generic/libffi", + "dep:generic/openssl", + "dep:generic/libffi", ] -SciPy 1.10: +``PKG-INFO`` / ``METADATA`` content: N/A. + +SciPy 1.10 +'''''''''' + +``pyproject.toml`` content: .. code:: toml [external] build-requires = [ - "virtual:compiler/c", - "virtual:compiler/cpp", - "virtual:compiler/fortran", - "pkg:generic/ninja", - "pkg:generic/pkg-config", + "dep:virtual/compiler/c", + "dep:virtual/compiler/cpp", + "dep:virtual/compiler/fortran", + "dep:generic/ninja", + "dep:generic/pkg-config", ] host-requires = [ - "virtual:interface/blas", - "virtual:interface/lapack", # >=3.7.1 (can't express version ranges with PURL yet) + "dep:virtual/interface/blas", + "dep:virtual/interface/lapack@>=3.7.1", ] -Pillow 10.1.0: +``PKG-INFO`` / ``METADATA`` content: N/A. + +Pillow 10.1.0 +''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] build-requires = [ - "virtual:compiler/c", + "dep:virtual/compiler/c", ] host-requires = [ - "pkg:generic/libjpeg", - "pkg:generic/zlib", + "dep:generic/libjpeg", + "dep:generic/zlib", ] [external.optional-host-requires] extra = [ - "pkg:generic/lcms2", - "pkg:generic/freetype", - "pkg:generic/libimagequant", - "pkg:generic/libraqm", - "pkg:generic/libtiff", - "pkg:generic/libxcb", - "pkg:generic/libwebp", - "pkg:generic/openjpeg", # add >=2.0 once we have version specifiers - "pkg:generic/tk", + "dep:generic/lcms2", + "dep:generic/freetype", + "dep:generic/libimagequant", + "dep:generic/libraqm", + "dep:generic/libtiff", + "dep:generic/libxcb", + "dep:generic/libwebp", + "dep:generic/openjpeg@>=2.0", + "dep:generic/tk", ] +``PKG-INFO`` / ``METADATA`` content: N/A. -NAVis 1.4.0: +NAVis 1.4.0 +''''''''''' + +``pyproject.toml`` content: .. code:: toml @@ -453,52 +700,107 @@ NAVis 1.4.0: [external] build-requires = [ - "pkg:generic/XCB; platform_system=='Linux'", + "dep:generic/XCB; platform_system=='Linux'", ] [external.optional-dependencies] nat = [ - "pkg:cran/nat", - "pkg:cran/nat.nblast", + "dep:cran/nat", + "dep:cran/nat.nblast", ] -Spyder 6.0: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Provides-External-Extra: nat + Requires-External-Dep: dep:cran/nat; extra == 'nat' + Requires-External-Dep: dep:cran/nat.nblast; extra == 'nat' + +Spyder 6.0 +'''''''''' + +``pyproject.toml`` content: .. code:: toml [external] dependencies = [ - "pkg:cargo/ripgrep", - "pkg:cargo/tree-sitter-cli", - "pkg:golang/github.com/junegunn/fzf", + "dep:cargo/ripgrep", + "dep:cargo/tree-sitter-cli", + "dep:golang/github.com/junegunn/fzf", ] -jupyterlab-git 0.41.0: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:cargo/ripgrep + Requires-External-Dep: dep:cargo/tree-sitter-cli + Requires-External-Dep: dep:golang/github.com/junegunn/fzf + +jupyterlab-git 0.41.0 +''''''''''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] dependencies = [ - "pkg:generic/git", + "dep:generic/git", ] [external.optional-build-requires] dev = [ - "pkg:generic/nodejs", + "dep:generic/nodejs", ] -PyEnchant 3.2.2: +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:generic/git + +PyEnchant 3.2.2 +''''''''''''''' + +``pyproject.toml`` content: .. code:: toml [external] dependencies = [ - # libenchant is needed on all platforms but only vendored into wheels on - # Windows, so on Windows the build backend should remove this external - # dependency from wheel metadata. - "pkg:github/AbiWord/enchant", + # libenchant is needed on all platforms but vendored into wheels + # distributed on PyPI for Windows. Hence choose to encode that in + # the metadata. Note: there is no completely unambiguous way to do + # this; another choice is to leave out the environment marker in the + # source distribution and either live with the unnecessary ``METADATA`` + # entry in the distributed Windows wheels, or to apply a patch to this + # metadata when building those wheels. + "dep:github/AbiWord/enchant; platform_system!='Windows'", ] +``PKG-INFO`` / ``METADATA`` content: + +.. code:: + + Requires-External-Dep: dep:github/AbiWord/enchant; platform_system!="Windows" + +With dependency groups +'''''''''''''''''''''' + +``pyproject.toml`` content: + +.. code:: toml + + [external.dependency-groups] + dev = [ + "dep:generic/catch2", + "dep:generic/valgrind", + ] + +``PKG-INFO`` / ``METADATA`` content: N/A. Backwards Compatibility ======================= @@ -507,6 +809,15 @@ There is no impact on backwards compatibility, as this PEP only adds new, optional metadata. In the absence of such metadata, nothing changes for package authors or packaging tooling. +The only change introduced in this PEP that has impact on existing projects is the +deprecation of the ``External-Requires`` Core Metadata field. We estimate the impact +of this deprecation to be negligible, given the its low penetration in the ecosystem +(see Rationale). + +The field will still be recognized by existing tools such as `setuptools-ext`_ +but its usage will be discouraged in the `Python Packaging User Guide`_, similar to +what is done for obsolete fields like ``Requires`` (deprecated in favor of +``Requires-Dist``). Security Implications ===================== @@ -542,10 +853,12 @@ there will not be code implementing the metadata spec as a whole. However, there are parts that do have a reference implementation: 1. The ``[external]`` table has to be valid TOML and therefore can be loaded - with ``tomllib``. + with ``tomllib``. This table can be further processed with the + `pyproject-external`_ package, demonstrated below. 2. The PURL specification, as a key part of this spec, has a Python package with a reference implementation for constructing and parsing PURLs: - `packageurl-python`_. + `packageurl-python`_. This package is wrapped in `pyproject-external`_ + to provide DepURL-specific validation and handling. There are multiple possible consumers and use cases of this metadata, once that metadata gets added to Python packages. Tested metadata for all of the @@ -554,6 +867,64 @@ wheels can be found in `rgommers/external-deps-build`_. This metadata has been validated by using it to build wheels from sdists patched with that metadata in clean Docker containers. +Example +------- + +Given a ``pyproject.toml`` with this ``[external]`` table: + +.. code-block:: toml + + [external] + build-requires = [ + "dep:virtual/compiler/c", + "dep:virtual/compiler/rust", + "dep:generic/pkg-config", + ] + host-requires = [ + "dep:generic/openssl", + "dep:generic/libffi", + ] + +You can use ``pyproject_external.External`` to parse it and manipulate it: + +.. code-block:: python + + >>> from pyproject_external import External + >>> external = External.from_pyproject_path("./pyproject.toml") + >>> external.validate() + >>> external.to_dict() + {'external': {'build_requires': ['dep:virtual/compiler/c', 'dep:virtual/compiler/rust', 'dep:generic/pkg-config'], 'host_requires': ['dep:generic/openssl', 'dep:generic/libffi']}} + >>> external.build_requires + [DepURL(type='virtual', namespace='compiler', name='c', version=None, qualifiers={}, subpath=None), DepURL(type='virtual', namespace='compiler', name='rust', version=None, qualifiers={}, subpath=None), DepURL(type='generic', namespace=None, name='pkg-config', version=None, qualifiers={}, subpath=None)] + >>> external.build_requires[0] + DepURL(type='virtual', namespace='compiler', name='c', version=None, qualifiers={}, subpath=None) + +Note the proposed ``[external]`` table was well-formed. With invalid contents such as: + +.. code-block:: toml + + [external] + build-requires = [ + "dep:this-is-missing-the-type", + "pkg:not-a-dep-url" + ] + +You would fail the validation: + +.. code-block:: python + + >>> external = External.from_pyproject_data( + { + "external": { + "build_requires": [ + "dep:this-is-missing-the-type", + "pkg:not-a-dep-url" + ] + } + } + ) + ValueError: purl is missing the required type component: 'dep:this-is-missing-the-type'. + Rejected Ideas ============== @@ -565,84 +936,100 @@ There are non-Python packages which are packaged on PyPI, such as Ninja, patchelf and CMake. What is typically desired is to use the system version of those, and if it's not present on the system then install the PyPI package for it. The authors believe that specific support for this scenario is not -necessary (or too complex to justify such support); a dependency provider for -external dependencies can treat PyPI as one possible source for obtaining the -package. +necessary (or at least, too complex to justify such support); a dependency +provider for external dependencies can treat PyPI as one possible source for +obtaining the package. An example mapping for this use case is proposed in +:pep:`804`. Using library and header names as external dependencies ------------------------------------------------------- A previous draft PEP (`"External dependencies" (2015) `__) proposed using specific library and header names as external dependencies. This -is too granular; using package names is a well-established pattern across -packaging ecosystems and should be preferred. +is both too granular, and insufficient (e.g., headers are often unversioned; +multiple packages may provide the same header or library). Using package names +is a well-established pattern across packaging ecosystems and should be +preferred. +Splitting host dependencies with explicit ``-dev`` or ``-devel`` suffixes +------------------------------------------------------------------------- -Open Issues -=========== +This convention is not consistent across packaging ecosystems, nor commonly +accepted by upstream package authors. Since the need for explicit control +(e.g., installing headers when a package is used as a runtime rather than a +build-time dependency) is quite niche and we don't want to add design +complexity without enough clear use cases, we have chosen to rely solely on the +``build``, ``host`` and ``run`` category split, with tools being in charge of +which category applies to each case in a context-dependent way. + +If this proves to be insufficient, a future PEP could use the URL qualifier +features present in the PURL schema (``?key=value``) to implement the necessary +adjustments. This can be done in a backwards compatible fashion. + +Identifier indirections +----------------------- + +Some ecosystems exhibit methods to select packages based on parametrized +functions like ``cmake("dependency")`` or ``compiler("language")``, which +return package names based on some additional context or configuration. This +feature is arguably not very common and, even when present, rarely used. +Additionally, its dynamic nature makes it prone to changing meaning over time, +and relying on specific build systems for the name resolution is in general not +a good idea. + +The authors prefer static identifiers that can be mapped explicitly via well +known metadata (e.g., as proposed in :pep:`804`). -Version specifiers for PURLs ----------------------------- - -Support in PURL for version expressions and ranges is still pending. The pull -request at `vers implementation for PURL`_ seems close to being merged, at -which point this PEP could adopt it. - -Versioning of virtual dependencies ----------------------------------- - -Once PURL supports version expressions, virtual dependencies can be versioned -with the same syntax. It must be better specified however what the version -scheme is, because this is not as clear for virtual dependencies as it is for -PURLs (e.g., there can be multiple implementations, and abstract interfaces may -not be unambiguously versioned). E.g.: - -- OpenMP: has regular ``MAJOR.MINOR`` versions of its standard, so would look - like ``>=4.5``. -- BLAS/LAPACK: should use the versioning used by `Reference LAPACK`_, which - defines what the standard APIs are. Uses ``MAJOR.MINOR.MICRO``, so would look - like ``>=3.10.0``. -- Compilers: these implement language standards. For C, C++ and Fortran these - are versioned by year. In order for versions to sort correctly, we choose to - use the full year (four digits). So "at least C99" would be ``>=1999``, and - selecting C++14 or Fortran 77 would be ``==2014`` or ``==1977`` respectively. - Other languages may use different versioning schemes. These should be - described somewhere before they are used in ``pyproject.toml``. - -A logistical challenge is where to describe the versioning - given that this -will evolve over time, this PEP itself is not the right location for it. -Instead, this PEP should point at that (to be created) location. - -Who defines canonical names and canonical package structure? ------------------------------------------------------------- - -Similarly to the logistics around versioning is the question about what names -are allowed and where they are described. And then who is in control of that -description and responsible for maintaining it. Our tentative answer is: there -should be a central list for virtual dependencies and ``pkg:generic`` PURLs, -maintained as a PyPA project. See -https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/62. -TODO: once that list/project is prototyped, include it in the PEP and close -this open issue. - -Syntax for virtual dependencies -------------------------------- - -The current syntax this PEP uses for virtual dependencies is -``virtual:type/name``, which is analogous to but not part of the PURL spec. -This open issue discusses supporting virtual dependencies within PURL: -`purl-spec#222 `__. - -Should a ``host-requires`` key be added under ``[build-system]``? ------------------------------------------------------------------ +Ecosystems that do implement these indirections can use them to support the +infrastructure designed to generate the mappings proposed in :pep:`804`. + +Adding a ``host-requires`` key under ``[build-system]`` +------------------------------------------------------- Adding ``host-requires`` for host dependencies that are on PyPI in order to better support name mapping to other packaging systems with support for -cross-compiling may make sense. -`This issue `__ tracks this topic -and has arguments in favor and against adding ``host-requires`` under +cross-compiling seems useful in principle, for the same reasons as this PEP +adds a ``host-requires`` under the ``[external]`` table. However, it isn't +necessary to include in this PEP, and hence the authors prefer to keep the +scope of this PEP limited - a future PEP on cross compilation may want to +tackle this. `This issue `__ +contains more arguments in favor and against adding ``host-requires`` under ``[build-system]`` as part of this PEP. +Reusing the ``Requires-External`` field in Core Metadata +-------------------------------------------------------- + +The `Core Metadata`_ specification contains one relevant field, namely +``Requires-External``. While at first sight it would be a good candidate to +record the ``external.dependencies`` table, the authors have decided to not +re-use this field to propagate the external runtime dependencies metadata. + +The ``Requires-External`` field has very loosely defined semantics as of +version 2.4. Essentially: ``name [(version)][; environment marker]`` (with +square brackets denoting optional fields). It is not defined what valid strings +for ``name`` are; the example in the specification uses both "C" as a language +name, and "libpng" as a package name. Tightening up the semantics would be +backwards incompatible, and leaving it as is seems unsatisfactory. DepURLs +would need to be decomposed to fit in this syntax. + +Allowing use of ecosystem-specific version comparison semantics +--------------------------------------------------------------- + +There are cases, in particular when dealing with pre-releases, where PEP 440 +semantics for version comparisons don't quite work. For example, ``1.2.3a`` may +indicate a release subsequent to ``1.2.3`` rather than an alpha version. To +handle such cases correctly, it would be necessary to allow arbitrary +versioning schemes. The authors of this PEP consider the added value of +allowing that is not justified by the additional complexity. If desired, a +package author can use either a code comment or the ``qualifier`` field of a +DepURL (see the Versioning section under Rationale) to capture this level of +detail. + +Open Issues +=========== + +None at this time. + References ========== @@ -663,9 +1050,9 @@ References .. [#pypackaging-native-cross] pypackaging-native: "Cross compilation" https://pypackaging-native.github.io/key-issues/cross_compilation/ -* The "``pkgconfig`` specification as an - alternative to ``ctypes.util.find_library``" thread (2023, Discourse): - https://discuss.python.org/t/pkgconfig-specification-as-an-alternative-to-ctypes-util-find-library/31379 +.. [#pkgconfig-and-ctypes-findlibrary] The "``pkgconfig`` specification as an + alternative to ``ctypes.util.find_library``" thread (2023, Discourse): + https://discuss.python.org/t/pkgconfig-specification-as-an-alternative-to-ctypes-util-find-library/31379 Copyright @@ -676,11 +1063,14 @@ CC0-1.0-Universal license, whichever is more permissive. .. _PyPI: https://pypi.org -.. _core metadata: https://packaging.python.org/specifications/core-metadata/ +.. _Core Metadata: https://packaging.python.org/specifications/core-metadata/ .. _setuptools: https://setuptools.readthedocs.io/ .. _setuptools metadata: https://setuptools.readthedocs.io/en/latest/setuptools.html#metadata .. _SPDX: https://spdx.dev/ .. _PURL: https://github.com/package-url/purl-spec/ +.. _version specifier: https://packaging.python.org/en/latest/specifications/version-specifiers/ +.. _dependency specifiers: https://packaging.python.org/en/latest/specifications/dependency-specifiers/ +.. _dependency groups specification: https://packaging.python.org/en/latest/specifications/dependency-groups/ .. _packageurl-python: https://pypi.org/project/packageurl-python/ .. _vers: https://github.com/package-url/purl-spec/blob/version-range-spec/VERSION-RANGE-SPEC.rst .. _vers implementation for PURL: https://github.com/package-url/purl-spec/pull/139 @@ -695,5 +1085,15 @@ CC0-1.0-Universal license, whichever is more permissive. .. _auditwheel: https://github.com/pypa/auditwheel .. _delocate: https://github.com/matthew-brett/delocate .. _delvewheel: https://github.com/adang1345/delvewheel +.. _verspurl: https://github.com/package-url/purl-spec/issues/386 .. _rgommers/external-deps-build: https://github.com/rgommers/external-deps-build +.. _pyproject-external: https://github.com/jaimergp/pyproject-external .. _Reference LAPACK: https://github.com/Reference-LAPACK/lapack +.. _setuptools-ext: https://pypi.org/project/setuptools-ext/ +.. _PyPI JSON API responses: https://docs.pypi.org/api/json/ +.. _pypa/hatch#1712: https://github.com/pypa/hatch/issues/1712 +.. _pypa/flit#353: https://github.com/pypa/flit/issues/353 +.. _pypa/setuptools#4220: https://github.com/pypa/setuptools/discussions/4220#discussioncomment-8930671 +.. _poetry-external-dependencies: https://pypi.org/project/poetry-external-dependencies/ +.. _PyO3/maturin@5b0e4808: https://github.com/PyO3/maturin/commit/5b0e4808bb8852fe796cd2848932a35fbb14de8b +.. _elfdeps: https://github.com/python-wheel-build/elfdeps/