Why does Poetry select incorrect dependencies when pip does not?

[johnthagen]

For the full details of this issue with Open3D 0.16.1 + Mac + Poetry, see:

• Open3d 0.16.1 fails to install correct dependencies under Poetry isl-org/Open3D#5747

But what confuses me is that this issue does not affect pip on the same machine with the version of Python / Open3d.

If I create a bare venv and pip install open3d==0.16.1 into it, I get these dependencies:

pip list
Package              Version
-------------------- -------
addict               2.4.0
attrs                22.1.0
click                8.1.3
ConfigArgParse       1.5.3
contourpy            1.0.6
cycler               0.11.0
dash                 2.7.0
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table           5.0.0
fastjsonschema       2.16.2
Flask                2.2.2
fonttools            4.38.0
itsdangerous         2.1.2
Jinja2               3.1.2
joblib               1.2.0
jsonschema           4.17.3
jupyter_core         5.1.0
kiwisolver           1.4.4
MarkupSafe           2.1.1
matplotlib           3.6.2
nbformat             5.5.0
numpy                1.23.5
open3d               0.16.1
packaging            21.3
pandas               1.5.2
Pillow               9.3.0
pip                  22.2.2
platformdirs         2.5.4
plotly               5.11.0
pyparsing            3.0.9
pyquaternion         0.9.9
pyrsistent           0.19.2
python-dateutil      2.8.2
pytz                 2022.6
PyYAML               6.0
scikit-learn         1.1.3
scipy                1.9.3
setuptools           65.4.1
six                  1.16.0
tenacity             8.1.0
threadpoolctl        3.1.0
tqdm                 4.64.1
traitlets            5.6.0
Werkzeug             2.2.2

But if I lock Open3D 0.16.1 using Poetry:

[tool.poetry.dependencies]
python = "^3.10, <3.11"
open3d = "0.16.1"

I get a much smaller list:

$ pip list
Package              Version
-------------------- -------
attrs                22.1.0
click                8.1.3
ConfigArgParse       1.5.3
dash                 2.7.0
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table           5.0.0
fastjsonschema       2.16.2
Flask                2.2.2
itsdangerous         2.1.2
Jinja2               3.1.2
jsonschema           4.17.3
jupyter_core         5.1.0
MarkupSafe           2.1.1
nbformat             5.5.0
numpy                1.23.5
open3d               0.16.1
pip                  22.3.1
platformdirs         2.5.4
plotly               5.11.0
pyrsistent           0.19.2
setuptools           65.6.3
tenacity             8.1.0
traitlets            5.6.0
Werkzeug             2.2.2
wheel                0.38.4

I know Poetry uses a JSON API and I think that pip uses something else (looking at the wheels themselves)?

1. Why do these tools resolve differently?
2. Is this an actual problem upstream in Open3d requirements?
3. Is this some symptom of an issue with how packages are registered to PyPI and Poetry is getting non-exhaustive data from the API?

[neersighted]

Poetry uses the PyPI JSON API when available to get requirements for a package (e.g. https://pypi.org/pypi/open3d/0.16.1/json). This API uses the metadata of the first wheel to be uploaded. In general, Poetry does not (currently) support wheels with disparate requirements, which is not necessary in 99% of Python packaging scenarios due to the richness of PEP 508 markers.

I have not figured out which package in the tree is providing inconsistent requirements, but that is probably what is happening, and the best place to start figuring out who is distributing wheels incompatible with Poetry.

[johnthagen]

I brought this up on the Packaging discourse for greater visibility:

• https://discuss.python.org/t/pep-rfc-python-package-index-warehouse-json-api-v1/9205/23

[johnthagen]

PyPA response, in case someone else stumbles across this:

• https://discuss.python.org/t/pep-rfc-python-package-index-warehouse-json-api-v1/9205/24

The simple API in theory allows you to fetch metadata per-wheel. In practice, Warehouse doesn’t yet expose that data, but it could. So if that’s what you want, it’s just a matter of waiting for that to be implemented. This is the tracking issue for that work.

The JSON API is fundamentally flawed, as it exposes metadata at the project/version level, and in fact, the way metadata is defined, it’s not possible to say what the dependencies of “X version 1.0” are - they can differ between builds.

It’s far easier to manage dependencies if you ignore this issue, and assume that the same dependencies are valid for all builds of a given project/version, but it simply isn’t true. We’re slowly trying to move projects to a position where we could feasibly make a rule that enforces that, but even if we do, there will still be older projects that won’t follow that rule, and it’ll be many years before we can ignore those 🙁

Without a change to the way metadata is standardised, “an accurate cross-platform list of dependencies” isn’t an achievable goal, except for projects that self-impose that restriction (by use of markers and similar techniques).

[apockill]

Hi, I'm curious if the conclusion is that this won't be looked into further. It looks like this problem is still an issue with open3d's last few versions since 0.16.0.

At the moment I'm able to pip install open3d==0.18.0 but unable to poetry add open3d==0.18.0 and have the latter work.

[johnthagen]

@apockill One temporary workaround is to manually list the missing dependencies in your own pyproject.toml:

[tool.poetry.dependencies]
open3d = "*"

# TODO: Remove these when Open3D releases >0.18.0 with ML dependencies properly set that is
#   compatible with Poetry. https://github.com/isl-org/Open3D/issues/5747 
#   https://github.com/isl-org/Open3D-ML/blob/main/requirements.txt
addict = "*"
pillow = ">=9.3.0"
matplotlib = ">=3"
numpy = ">1.18"
pandas = ">=1.0"
pyyaml = ">=5.4.1"
scikit-learn = ">=0.21"
tqdm = "*"
pyquaternion = "*"

It's not ideal, but it does work.

[apockill]

Got it, thank you for the response!