Extras names are incorrectly normalized in poetry-core >= 1.3.0

[0xallie]

• Poetry version: N/A (poetry-core 1.3.2)
• Python version: 3.10.7
• OS version and name: Debian 11
• pyproject.toml: https://gist.github.com/nyuszika7h/b3929c066e2f19e85593c573bf49c296

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

This appears to be a regression introduced in poetry-core 1.3.0. I'm opening the issue here because issues are closed on the poetry-core repo.

When using poetry-core >= 1.3.0, names of extras are incorrectly normalized to change _ to -, causing their installation to fail:

❯ pip install .
Processing /home/nyuszika7h/test
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests[use-chardet-on-py3]<3.0.0,>=2.28.1 in ./.venv/lib/python3.10/site-packages (from test==0.1.0) (2.28.1)
WARNING: requests 2.28.1 does not provide the extra 'use-chardet-on-py3'
Requirement already satisfied: charset-normalizer<3,>=2 in ./.venv/lib/python3.10/site-packages (from requests[use-chardet-on-py3]<3.0.0,>=2.28.1->test==0.1.0) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.10/site-packages (from requests[use-chardet-on-py3]<3.0.0,>=2.28.1->test==0.1.0) (2022.9.24)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.10/site-packages (from requests[use-chardet-on-py3]<3.0.0,>=2.28.1->test==0.1.0) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.venv/lib/python3.10/site-packages (from requests[use-chardet-on-py3]<3.0.0,>=2.28.1->test==0.1.0) (1.26.12)
Building wheels for collected packages: test
  Building wheel for test (pyproject.toml) ... done
  Created wheel for test: filename=test-0.1.0-py3-none-any.whl size=827 sha256=ad5626a8f9c6f9ceafa81586876887f6ffb6df743a34fb0ba50ecfd54371ad89
  Stored in directory: /tmp/pip-ephem-wheel-cache-53k7_wx9/wheels/ff/6c/d3/80cb9d9d69b88685ef924814b380c07445b80a9c85a2cb1b54
Successfully built test
Installing collected packages: test
Successfully installed test-0.1.0

With poetry-core 1.2.0:

❯ pip install .
Processing /home/nyuszika7h/test
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests[use_chardet_on_py3]<3.0.0,>=2.28.1 in /home/nyuszika7h/.local/share/pyenv/versions/3.10.7/lib/python3.10/site-packages (from test==0.1.0) (2.28.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/nyuszika7h/.local/share/pyenv/versions/3.10.7/lib/python3.10/site-packages (from requests[use_chardet_on_py3]<3.0.0,>=2.28.1->test==0.1.0) (2022.9.24)
Requirement already satisfied: idna<4,>=2.5 in /home/nyuszika7h/.local/share/pyenv/versions/3.10.7/lib/python3.10/site-packages (from requests[use_chardet_on_py3]<3.0.0,>=2.28.1->test==0.1.0) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in /home/nyuszika7h/.local/share/pyenv/versions/3.10.7/lib/python3.10/site-packages (from requests[use_chardet_on_py3]<3.0.0,>=2.28.1->test==0.1.0) (2.1.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/nyuszika7h/.local/share/pyenv/versions/3.10.7/lib/python3.10/site-packages (from requests[use_chardet_on_py3]<3.0.0,>=2.28.1->test==0.1.0) (1.26.12)
Collecting chardet<6,>=3.0.2
  Using cached chardet-5.0.0-py3-none-any.whl (193 kB)
Building wheels for collected packages: test
  Building wheel for test (pyproject.toml) ... done
  Created wheel for test: filename=test-0.1.0-py3-none-any.whl size=823 sha256=30ec1efd1fc1ca60f9ced0c8850a14acc5dfdc4c98dc2fa6840c4d6153dfe57f
  Stored in directory: /tmp/pip-ephem-wheel-cache-z_hwotnm/wheels/ff/6c/d3/80cb9d9d69b88685ef924814b380c07445b80a9c85a2cb1b54
Successfully built test
Installing collected packages: chardet, test
Successfully installed chardet-5.0.0 test-0.1.0

[neersighted]

The issue is not the normalization -- the issue is that the other side of the comparison is not normalized when looking for extras, as best I can tell from a quick skim. The normalization is specified in PEP 685.

PTAL @dimbleby

[neersighted]

Ah, I just processed that this was pip install and not poetry install -- I think this might be a pip bug. They are required by spec to normalize extra names before making a comparison -- Poetry is writing the canonical name, which by spec should be considered equivalent.

[dimbleby]

yep, looks like basically a duplicate of python-poetry/poetry-plugin-export#150

here's a pip bug requesting pep 685 be implemented pypa/pip#11445

I've not looked at the pip code but I can't imagine that the part of pep685 which says "normalize extra names before comparison" can be very hard to do, perhaps someone who is affected by this would like to put together an MR over there

[dimbleby]

looks as though this was implemented in packaging some months ago - pypa/packaging#545 - but the subsequent release has gotten a bit stalled - pypa/packaging#569

However updates in the last couple of days suggest that it's coming imminently, and I'd guess that pip will pick that up in their next release.

So IMO poetry should hold firm here. Reverting the change, apart from being tedious, would reintroduce the bugs that it fixed (#6321) - ie it's not obviously making things better - and it looks reasonable to expect that the ecosystem is close to catching up with the specs.

[ofek]

Could do: https://hatch.pypa.io/latest/config/metadata/#allowing-ambiguous-features

[dimbleby]

It's taken somewhat longer than I'd guessed it would, but packaging 22.0 has now been released

So I guess that a pip release picking that up will happen in due course, and pip should soon agree with poetry - or rather with PEP685 - about how extras are normalized.

[dimbleby]

well it really is taking a while for pip to implement pep 685, pypa/pip#11445 seems to be harder than I'd have guessed.

nevertheless there is no poetry bug here and no intention of changing poetry behaviour, might as well close this one out I think