Deprecate non-PEP 625 sdist support?

[woodruffw]

What's the problem this feature will solve?

Hello pip maintainers!

TL;DR: I'd like to propose removing support for non-PEP 625 source distributions, as part of aligning pip (and other standard packaging infrastructure) with the packaging PEPs/living PyPA standards.

This would be a nontrivial change, and so I imagine it would require a full (and long) deprecation cycle. I've put some notes below about how I think this could be done gracefully to avoid breakage against the majority of packages currently on PyPI (which potentially have a long tail of non-PEP 625 conformance).

For additional context:

• PyPI has fully supported PEP 625 as of this past October: Support PEP 625 (Filename of a Source Distribution) pypi/warehouse#12245

Describe the solution you'd like

I'd like pip to (eventually) reject sdists that don't conform to PEP 625. That means:

• No more support for non-standard sdist archive forms (zips, .tar.xz, etc).
• Enforcing that the {distribution} is encoded per 625 (which stipulates normalization like in the wheel spec)
• Enforcing that the {version} is a valid and normalized PEP 440 version.

These requirements are tracked in the living version of PEP 625 as well:

https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-distribution-file-format

One significant hangup with this is the long tail of sdists on PyPI that aren't necessarily conformant: a significant number of users install via sdists, and a lot of sdists either predate PEP 625 or were uploaded to PyPI before strict enforcement.

Given that, I think a staggered deprecation cycle probably makes sense here. At a high-level (these specific timeframes may not make sense):

1. Begin emitting deprecation warnings for all sdists that don't match PEP 625
2. 12+ months: begin rejecting sdists that aren't valid tar+gz tarballs
3. 24+ months: begin rejecting sdists that don't have valid {distribution}-{version} schemes

The rationale for the above is that non-tar+gz sdists are probably much less common than ones with non-normalized names/versions, so it's probably fine to reject them sooner. I intend to run some statistics on a public dump of PyPI to confirm that.

Separately, it probably makes sense to have some kind of escape hatch for this behavior, e.g. for integrators who have archives that are sdist-shaped but can't be given PEP 625 filenames for whatever reason. Something like pip install --allow-invalid-sdist-name ... perhaps (that's probably not a good name, but just to show the idea).

Alternative Solutions

One alternative solution is to do nothing 🙂 -- on a basic level non-conforming sdists aren't a "big deal," since an sdist is just an archive with a build script. However, insofar as conformance with the PEPs/living PyPA standards is a long term goal for Python packaging, I think aiming to conform with this one is a good idea!

Another alternative solution is to only go part-ways here: instead of hard-rejecting invalid sdists, pip could choose to emit a warning instead (and never hard-reject). The upside to this is stability; the downside is that users will probably mostly ignore the warning.

Additional context

This piqued my interest because of astral-sh/uv#16911 -- uv (like pip) currently supports .tar.xz as well as xz-in-ZIP, despite neither being a standard-conformant distribution representation (for either sdists or wheels).

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[woodruffw]

CC @zanieb and @konstin 🙂

[zanieb]

There's also an argument beyond conformance, i.e., that removing support for, e.g., xz, reduces supply chain attack surfaces (which is the main reason we care in uv and is noted in the linked uv issue).

[woodruffw]

Yeah, thanks for calling that out! I buried that a bit in the lede by accident, but I think there's a significant win to be had in reducing the number of implicitly valid archive format/compressor combinations.

[pfmoore]

In general terms, I'm very much in favour of this. But I think the biggest problem is the long tail of non-standard sdists. And it's not just on PyPI - we have no way of knowing how many in-house private packages exist which don't use the standard naming.

One thing which would be very interesting is an analysis of how many packages exist on PyPI where the current version still uses a non-standard name. Presumably at some point if we're talking about installers not supporting non-standard names, we'd want to start deleting them from PyPI - and if we did that, how many packages would be completely removed as a result?

[woodruffw]

One thing which would be very interesting is an analysis of how many packages exist on PyPI where the current version still uses a non-standard name.

Yeah, I think this would be a leading indicator for whether this'll actually be workable. I'm going to try and pull that from https://github.com/sethmlarson/pypi-data today 🙂

Edit: I realized pypi-data doesn't have this, since it doesn't track sdists separately.

Presumably at some point if we're talking about installers not supporting non-standard names, we'd want to start deleting them from PyPI - and if we did that, how many packages would be completely removed as a result?

I think we'd probably want to treat this similarly to .eggs and other deprecations, where PyPI wouldn't delete anything but newer clients would just stop supporting them (and would warn appropriately when the user tries to install directly). That way legacy users can still continue to use them, but the ecosystem as a whole can move forwards.

[lengau]

Removing support for xz-compressed tarballs (and possibly bz2-compressed ones) would break workflows for people using packages from linux distributions or who need distro+version specific packages.

For my case in particular, we currently need access to the python-apt source packages, which is distributed by the Debian and Ubuntu teams as xz-compressed tarballs. (Likely there are similar packages that affect people on Red Hat or SuSE based distributions too.) Given that these distributions are supported well beyond the lifecycle of a python version, even if the relevant source packages were swapped to use gzip today, there would be a very long tail where people developing on/for still-supported distros would break when using a newer pip (or uv), likely leading to people sticking with an older pip even for virtual environments.

[woodruffw]

Removing support for xz-compressed tarballs (and possibly bz2-compressed ones) would break workflows for people using packages from linux distributions or who need distro+version specific packages.

Could you say a bit more about the workflow here? My understanding is that distributions generally provide Python packages using their native packaging format. In other words, most users are supposed to do something like apt install -y python-apt to install this package, rather than fetch it directly from an archive.

I'm not the decision maker here, but I think there are a handful of mitigating considerations that will make this less troublesome for distributions:

1. Per above, this won't affect distribution level packages; it's only for Python packaging, which has its own standards/metadata/etc;
2. If this happens, the deprecation period probably will be extensive (it took 5 years to get from PEP 625's acceptance to adoption by PyPI; I hope this wouldn't be that long but such timelines are not unprecedented);
3. On top of the above, I think the proposed escape hatch (some kind of flag or environment variable to allow non-PEP 625 sdists) would enable the use case laid out;
4. Finally, distributions with LTS policies could always flip that default or carry patches for the version of pip they distribute. This definitely wouldn't be ideal, but I think it'd be similar to other cases where Python packaging has undergone standards changes that distributions may not want to pursue on the same timeline.

[pfmoore]

Removing support for xz-compressed tarballs (and possibly bz2-compressed ones) would break workflows for people using packages from linux distributions or who need distro+version specific packages.

If these types of package are needed as sdists (as opposed to simple source trees), then someone needs to propose that those formats get added to the packaging standards. At the moment, there is no requirement for an installer to support those formats. I don't expect .xz support to be that controversial (I'd imagine .bz2 support might get more pushback), so there's no reason to believe that standardising is "too hard".

Expecting the packaging ecosystem to continue indefinitely supporting formats that aren't defined by standards isn't really a reasonable position to take, IMO.

For my case in particular, we currently need access to the python-apt source packages

The .tar.xz files in that directory don't look remotely like sdists to me - they have the wrong normalisation and filename structure. They are almost certainly totally fine as archived source trees (so you can pip install https://ftp.debian.org/debian/pool/main/p/python-apt/python-apt_3.0.0.tar.xz) but expecting them to work as sdists (where you do pip install python-apt and expect pip to find that file) is unreasonable, IMO¹.

Footnotes

1. A standards-conforming installer would see that file as versionapt_3.0.0 of package python, and reject it as the version doesn't conform to the version number standard. ↩

[notatallshaw]

w.r.t to filename parsing and restrictions I would like to see that code come from packaging but I believe there are still open questions, or at least work still to do:

pypa/packaging#527
pypa/packaging#873

[woodruffw]

w.r.t to filename parsing and restrictions I would like to see that code come from packaging but I believe there are still open questions, or at least work still to do:

pypa/packaging#527

Hoisted upon my own petard...I forgot I filed that 4 years ago. I have some thoughts on disambiguating that API that I'll follow up on there.

[notatallshaw]

FYI, we use currently only use parse_sdist_filename from packaging in one location in pip, when we scan a local file directory for distributions and we filter to only filenames that are valid wheels or sdists filenames. I somewhat naively implemented this, not checking on the history or outstanding questions about the function.

[gpshead]

we have no way of knowing how many in-house private packages exist which don't use the standard naming

This is what a warning can be used to find out?

While I'd be surprised if anyone bothered using non-defaults for in-house things - the only feedback mechanism we've got there is making the code encourage people to speak up when they do so you can understand if they cannot simply recompress all of their in-house things in the supported format and if it is important to support that or not.