Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PEP 685: Copyedit and fix various minor issues following further changes #2428

Merged
merged 3 commits into from
Mar 16, 2022
Merged

PEP 685: Copyedit and fix various minor issues following further changes #2428

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d58495e
PEP 685: Update headers & fix reST syntax, links & punctuation
CAM-Gerlach Mar 14, 2022
09b5431
PEP 685: Fix grammar & clarity issues, improve phrasing & avoid rep
CAM-Gerlach Mar 14, 2022
d3a0a26
PEP 685: Restore citations as originally intended
CAM-Gerlach Mar 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 34 additions & 32 deletions pep-0685.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Mar-2022
Post-History: 08-Mar-2022
Post-History: `08-Mar-2022 <https://discuss.python.org/t/14141>`__


Abstract
Expand All @@ -28,13 +28,13 @@ name "must be a valid Python identifier".
letter, digit, or any one of ``.``, ``-``, or ``_`` after the initial character.
Otherwise, there is no other `PyPA specification
<https://packaging.python.org/en/latest/specifications/>`_
which outlines how extra names should be written or normalization for comparison.
which outlines how extra names should be written or normalized for comparison.
Due to the amount of packaging-related code in existence,
it is important to evaluate current practices by the community and
standardize on one that doesn't break most code, while being
standardize on one that doesn't break most existing code, while being
something tool authors can agree to following.

The issue of there being no standard was brought forward by an
The issue of there being no consistent standard was brought forward by an
`initial discussion <https://discuss.python.org/t/7614>`__
noting that the extra ``adhoc-ssl`` was not considered equal to the name
``adhoc_ssl`` by pip 22.
Expand All @@ -47,27 +47,29 @@ Rationale

re.sub(r"[-_.]+", "-", name).lower()

This collapses any run of the substitution character down to a single
character,
e.g. ``---`` gets collapsed down to ``-``.
This does **not** produce a valid Python identifier as specified by
This collapses any run of the characters ``-``, ``_`` and ``.``
down to a single ``-``.
For example, ``---`` ``.`` and ``__`` all get converted to just ``-``.
This does **not** produce a valid Python identifier, per
the core metadata 2.2 specification for extra names.

`Setuptools 60 does normalization <https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330>`__
`Setuptools 60 performs normalization <https://github.com/pypa/setuptools/blob/b2f7b8f92725c63b164d5776f85e67cc560def4e/pkg_resources/__init__.py#L1324-L1330>`__
via::

re.sub(r'[^A-Za-z0-9-.]+', '_', name).lower()

The use of an underscore/``_`` differs from PEP 503's use of a
hyphen/``-``.
Runs of ``.`` and ``-``, unlike PEP 503, do **not** get collapsed,
e.g. ``..`` stays the same.
The use of an underscore/``_`` differs from PEP 503's use of a hyphen/``-``,
and it also normalizes characters outside of those allowed by :pep`508`.
Runs of ``.`` and ``-``, unlike PEP 503, do **not** get normalized to one ``_``,
e.g. ``..`` stays the same. To note, this is inconsistent with this function's
docstring, which *does* specify that all non-alphanumeric characters
(which would include ``-`` and ``.``) are normalized and collapsed.

For pip 22, its
"extra normalisation behaviour is quite convoluted and erratic" [pip-erratic]_,
"extra normalisation behaviour is quite convoluted and erratic" [pip-erratic]_
and so its use is not considered.

.. [pip-erratic] https://discuss.python.org/t/what-extras-names-are-treated-as-equal-and-why/7614/10?
.. [pip-erratic] Tzu-ping Chung on Python Discourse <https://discuss.python.org/t/7614/10


Specification
Expand Down Expand Up @@ -96,39 +98,38 @@ name is provided as appropriate for the specified core metadata version.
If an older core metadata version is specified and the name would be
invalid with newer core metadata versions,
tools SHOULD warn the user.
Tools SHOULD warn users when an invalid extra name is read and not use
Tools SHOULD warn users when an invalid extra name is read and SHOULD not use
the name to avoid ambiguity.
Tools MAY raise an error instead of a warning when reading an
invalid name if they so desire.
invalid name, if they so desire.


Backwards Compatibility
=======================

Moving to :pep:`503` normalization and :pep:`508` name acceptance, it
Moving to :pep:`503` normalization and :pep:`508` name acceptance
allows for all preexisting, valid names to continue to be valid.

Based on research looking at a collection of wheels on PyPI [pypi-results]_,
the risk of extra name clashes is limited to 73 clashes when considering
even invalid names,
the risk of extra name clashes is limited to 73 instances when considering
all extras names on PyPI, valid or not (not just those within a single package)
while *only* looking at valid names leads to only 3 clashes:

1. dev-test: dev_test, dev-test, dev.test
2. dev-lint: dev-lint, dev.lint, dev_lint
3. apache-beam: apache-beam, apache.beam
* ``dev-test``: ``dev_test``, ``dev-test``, ``dev.test``
* ``dev-lint``: ``dev-lint``, ``dev.lint``, ``dev_lint``
* ``apache-beam``: ``apache-beam``, ``apache.beam``

By requiring tools writing core metadata to only record the normalized name,
the issue of preexisting, invalid extra names should be diminished over
time.
the issue of preexisting, invalid extra names should diminish over time.

.. [pypi-results] https://discuss.python.org/t/pep-685-comparison-of-extra-names-for-optional-distribution-dependencies/14141/17?u=brettcannon
.. [pypi-results] Paul Moore on Python Discourse https://discuss.python.org/t/14141/17


Security Implications
=====================

It is possible that for a distribution that has conflicting extra names, a
tool ends up installing distributions that somehow weaken the security
tool ends up installing dependencies that somehow weaken the security
of the system.
This is only hypothetical and if it were to occur,
it would probably be more of a security concern for the distributions
Expand All @@ -149,7 +150,7 @@ Reference Implementation

No reference implementation is provided aside from the code above,
but the expectation is the `packaging project`_ will provide a
function in its ``packaging.utils`` that will implement extra name
function in its ``packaging.utils`` module that will implement extra name
normalization.
It will also implement extra name comparisons appropriately.
Finally, if the project ever gains the ability to write out metadata,
Expand All @@ -162,11 +163,12 @@ Rejected Ideas
Using setuptools 60's normalization
-----------------------------------

Initially this PEP proposed following setuptools to try and minimize
backwards-compatibility issues.
But after checking various wheels on PyPI,
Initially, this PEP proposed using setuptools ``safe_extra()`` for normalization
to try to minimize backwards-compatibility issues.
However, after checking various wheels on PyPI,
it became clear that standardizing **all** naming on :pep:`508` and
:pep:`503` semantics was easier and better long-term.
:pep:`503` semantics was easier and better long-term,
while causing minimal backwards compatibility issues.


Open Issues
Expand Down