Dependency resolution using Azure Package Feed no longer operates correctly

[ARawles]

Description

When using an Azure Package Feed, poetry's dependency resolution fails to find the appropriate dependencies.

Azure Package Feed (APF) essentially acts as a PyPi mirror, allowing you to install directly from the package feed (as if you're installing from PyPi) but then provides the ability to block downloads of certain packages as well as add private packages to the repository.

Therefore, a common setup when using an APF is to use the feed as your "primary" source:

[[tool.poetry.source]]
name = "my-feed"
url = "https://pkgs.dev.azure.com/xxx/uuid/_packaging/my-python-feed/pypi/simple/"
priority = "primary"

When an install request reaches the APF, if the package hasn't already been used before, it will check PyPi for the package and copy it if it exists. The same is also true for new versions of packages.

Previously, the dependency resolution behaviour of PyPi and APF was identical. But now, more recent versions of certain packages no longer resolve correctly. Pydantic is maybe the best example of this.

Starting from Pydantic 2.7.0 (released 11/04/2024), poetry will no longer resolve the dependencies properly from an APF, but will correctly resolve them from PyPi. Example below:

Using APF as "primary"

(myvenv-py3.11):~/$ poetry debug resolve pydantic=="2.6.4"
Resolving dependencies... (1.1s)

Resolution results:

typing-extensions 4.11.0
annotated-types   0.6.0 
pydantic-core     2.16.3
pydantic          2.6.4 

(myvenv-py3.11):~/$ poetry debug resolve pydantic=="2.7.0"
Resolving dependencies... (0.1s)

Resolution results:

pydantic 2.7.0

Using PyPI as "primary"

(myvenv-py3.11):~/$ poetry debug resolve pydantic=="2.6.4"
Resolving dependencies... (1.1s)

Resolution results:

typing-extensions 4.11.0
annotated-types   0.6.0 
pydantic-core     2.16.3
pydantic          2.6.4 

(myvenv-py3.11):~/$ poetry debug resolve pydantic=="2.7.0"
Resolving dependencies... (0.5s)

Resolution results:

typing-extensions 4.11.0
annotated-types   0.6.0 
pydantic-core     2.18.1
pydantic          2.7.0

I'm not sure exactly what's caused this - if we look in the package entry in APF, the requirements are being listed correctly, they're just not being picked up by poetry:

Workarounds

Theoretically, one could use PyPi as the primary and then the APF as the supplemental, but this fundamentally defeats the purpose of using the APF in the first instance. It also leaves you open to various attack vectors.

Poetry Installation Method

pip

Operating System

ubuntu-22.04

Poetry Version

1.7.1

Poetry Configuration

cache-dir = "/home/user/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
repositories.my-feed.url = "https://pkgs.dev.azure.com/xxx/uuid/_packaging/my-python-feed/pypi/simple/"
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/user/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

[tool.poetry]
name = "example"

[tool.poetry.dependencies]
pydantic = "2.7.0"

[[tool.poetry.source]]
name = "my-feed"
url = "https://pkgs.dev.azure.com/xxx/uuid/_packaging/my-python-feed/pypi/simple/"
priority = "primary"

Poetry Runtime Logs

Loading configuration file /home/user/.config/pypoetry/config.toml
Adding repository my-feed (https://pkgs.dev.azure.com/xxx/uuid/_packaging/my-python-feed/pypi/simple/) and setting it as primary
Adding repository PyPI (https://pypi.org/simple/) and setting it as supplemental
Resolving dependencies...
   1: fact: example is 0.1.0
   1: derived: example
   1: fact: example depends on pydantic (2.7.0)
   1: selecting example (0.1.0)
   1: derived: example (==2.7.0)
Source (my-feed): 1 packages found for pydantic 2.7.0
   1: selecting pydantic (2.7.0)
   1: Version solving took 0.004 seconds.
   1: Tried 1 solutions.

Resolution results:

pydantic 2.7.0

[dimbleby]

duplicate #9130, #9191, #9195, #9244, #9266, #9272, #9288, #9294 - with same resolution

please close

[ARawles]

For those that don't want to read through the other issues, the fix is to make sure that both poetry and (in my case) pkginfo are up to date

[tomwatson1024]

Just to confirm - this issue is fixed in 1.8.3, but present in 1.8.2?

[radoering]

In Poetry 1.8.2, you have to make sure by yourself that a recent enough pkginfo version is installed:

• Poetry 1.8.2 requires pkginfo>=1.9.4,<2.0.0
• Poetry 1.8.3 requires pkginfo>=1.10,<2.0
• This issue occurs if you have pkginfo<1.10

In any case, you must clear your cache after experiencing this issue.

[Philaeux]

TLDR of commands to run to fix:

• update pkginfo pip pipx python -m pip install --upgrade pip pipx pkginfo
• update poetry pipx upgrade poetry
• clear cache poetry cache clear --all .
• rebuild graph of dependencies poetry update