Non-tagged Python builds use non-PEP 440 versions

[jaraco]

Sorry if this should be a new issue. I have a problem where the version of Python itself is causing an Invalid Version error. I install Python from the tip of 3.12, then:

% .tox/anypy/bin/python -c "import pkg_resources as p; p.load_entry_point('coverage', 'console_scripts', 'coverage')()"
Traceback (most recent call last):
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2711, in _dep_map
    return self.__dep_map
           ^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2826, in __getattr__
    raise AttributeError(attr)
AttributeError: _Distribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 522, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2855, in load_entry_point
    return ep.load()
           ^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2467, in load
    self.require(*args, **kwargs)
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2489, in require
    reqs = self.dist.requires(self.extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2746, in requires
    dm = self._dep_map
         ^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2713, in _dep_map
    self.__dep_map = self._filter_extras(self._build_dep_map())
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2728, in _filter_extras
    invalid_marker(marker) or not evaluate_marker(marker)
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 1415, in invalid_marker
    evaluate_marker(text)
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/__init__.py", line 1433, in evaluate_marker
    return marker.evaluate()
           ^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/markers.py", line 245, in evaluate
    return _evaluate_markers(self._markers, current_environment)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/markers.py", line 151, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/markers.py", line 109, in _eval_op
    return spec.contains(lhs, prereleases=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 565, in contains
    normalized_item = _coerce_version(item)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
              ^^^^^^^^^^^^^^^^
  File "/Users/nedbatchelder/coverage/trunk/.tox/anypy/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '3.12.0a5+'

% .tox/anypy/bin/python -m pip freeze --all
attrs==22.2.0
colorama==0.4.6
-e git+ssh://git@github.com/nedbat/coveragepy.git@1edd0608fac9ffadf24da72820572bec43d0c8fd#egg=coverage
distlib==0.3.6
exceptiongroup==1.1.0
execnet==1.9.0
filelock==3.9.0
flaky==3.7.0
hypothesis==6.68.0
importlib-metadata==6.0.0
iniconfig==2.0.0
packaging==23.0
pip==23.0
platformdirs==3.0.0
pluggy==1.0.0
pytest==7.2.1
pytest-xdist==3.2.0
setuptools==67.2.0
sortedcontainers==2.4.0
tomli==2.0.1
typing_extensions==4.4.0
virtualenv==20.19.0
wheel==0.38.4
zipp==3.12.1

Why is setuptools trying to parse the Python version?

There's a corresponding coverage.py issue about this.

Originally posted by @nedbat in pypa/setuptools#3772 (comment)

[jaraco]

I have a problem where the version of Python itself is causing an Invalid Version error.

Looking at the implementation of _build_dep_map, I suspect that some library has a dependency with a python_full_version marker, and when packaging attempts to parse the Python version (platform.python_version()), it's invalid (at least per PEP 440).

You can probably replicate the issue by installing packaging, then running:

>>> import packaging
>>> packaging.markers.Marker('python_full_version > "3.0"').evaluate()
True

I'm unsure if Python should use PEP 440 version numbers or if packaging should be updated to honor Python versions that aren't PEP 440 compliant, but this issue only happens to be loosely affiliated with this issue, which is about package versions.

[uranusjr]

Python has never formally followed PEP 440 (although most of the version numbers it used are compatible). Even if we start mandating PEP 440 for it, there are still much too many problematic version strings to deal with (considering redistributors probably have their own scheme). A separate parser for Python versions is probably more viable; IIRC Python has documentation on the rules somewhere, but we can ask the core devs for a concrete ruling if not.

[pradyunsg]

This was flagged to me on a private Discord today, and jotting down a summary/my takeaways from that discussion. There's three directions we can take here:

• Make packaging parse Python versions slightly differently.
• Change the definition of python_full_version to always generate valid PEP 440 versions.
• Change how Python picks in-development Python versions.

Broadly though, the underlying reason of why this was surfaced is the same as all other PEP 440 issues -- #530

---

For the first bullet:

I don't think this deviates much from PEP 440 TBH, and it's not clear that it's worth a compete new concept/mental model. Plus, it only affect development builds and patched copies of CPython. It's unclear that there's much value to changing that.

---

For the second bullet:

platform.python_version is documented as:

Returns the Python version as string 'major.minor.patchlevel'.

I think the expectation with PEP 508 is that all segments in the documentation would be numbers. We could just strip the trailing + and utilize that in this context and be fine.

---

For the third bullet:

There's also valid concern that people might be splitting the Python version on '.', so using a .devN suffix instead of the + suffix (i.e. 3.12.0a5+ vs 3.12.0a5.dev0). Folks seem to be open to the idea of changing the 3.12.0a5+ to 3.12.0a5+dev or something.

---

Adding cross-references mentioned by @hugovk and @Yhg1s:

• Python version ending with plus sign not compatible with PEP 508 python_full_version marker python/cpython#99968
• Poetry is incompatible with unstable-tagged Python builds (Invalid PEP 440 version: '3.Y.Z+') python-poetry/poetry#6925
• Python nightly tests fail: "InvalidVersion: Invalid version: '3.12.0a4+'" nedbat/coveragepy#1556

[CAM-Gerlach]

Not to bikeshed too early, but I would suggest something other than dev for the local identifier as +dev could be mistaken for the more conventional .devN (which would be the most "canonical"/semantically accurate way to do this, though +local is still syntactically valid of course which is the primary concern). So perhaps a different name should be preferred.