Optimize dependancies solving with C ecosystem library

[dev590t]

Is your feature request related to a problem? Please describe.

Dependancies solving is a high performance computing task. I'm never convinced that Python is the good language for such task. For some project, PDM has take more one hour to try to solve the dependancies, and it did not succeed.

Describe the solution you'd like

I appreciate there is a file pdm.lock, that allow developer to solve dependancies with other high performance tool, where PDM fails. After we can just use pdm install to install dependancies.

But I think that could be better if pdm is based on high performance dependancie solver with parallelization from C ecosystem.

[pawamoy]

This has been discussed on the Poetry tracker I think: solving Python dependencies is not a CPU- but an IO-bound task (network/disk and yes, some CPU), because the data isn't provided by the index since most of the packages still use dynamic setups and/or don't correctly declare their dependencies. Using C won't help at all 😕

UPDATE: high-level explanation here: https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow

[pawamoy]

By the way

that allow developer to solve dependancies with other high performance tool, where PDM fails.

do these tools actually exist, or are you just suggesting they could be created? Genuinely curious 🙂

[dev590t]

When I was studient, we has use Oz a logic programming language to solve the dependancies of package. But we need in first the metadata of all packages. But if the issue is from IO-bound task for pdm, this solution is not applicable.
But the reply of frostming below has make me think a hack: #1143 (reply in thread)

[frostming]

Yes, most of the time the solver is downloading distributions, extracting them, and reading the metadata. It is a heavy IO-bound task. C implementations don't help in this scenario.

AFAIK, conda is using mamba, a SAT solver implemented in C, with a carefully designed package server, which calculates and caches the dependency map on the server. But I don't think we can easily adopt it, since we have to stick to the python package index.

[pawamoy]

I don't think the dependency cache of mamba is compatible with the packages published on PyPI. Different artifacts, so different hashes, and possibly different dependencies. The Poetry project was planning at some point to create and manage such a cache so that dependency data is available quickly without having to actually download distributions.

I think it's still planned? python-poetry/poetry#1856

Development of a package registry to speed up dependency resolution of packages with missing metadata.

[dev590t]

I don't mean use directly the dependency cache of mamba. I mean use conda to solve the dependancy graph of the project, then compile the dependancy graph metadata from conda to pdm.lock or pyproject.toml. That should be possible if conda expose its project dependancies as pdm in a file (ie. pdm.lock for pdm). No?

[pawamoy]

Oh I see, Yes, interesting idea 🙂

[dev590t]

I have try my conda solution. conda package repo doesn't contain package that I want. This isn't really a good solution.

I have discovered python package manager pyflow. It seem it cache dependency information in a remote repo:
https://github.com/David-OConnor/pyflow#how-dependencies-are-resolved
https://github.com/David-OConnor/pyflow#dependency-cache-repo

Maybe pdm can try to use the remote cache repo of pyflow to speed up?

[frostming]

I think the simple self-hosted pydeps service is just a reliable version of PyPI JSON API: https://pypi.org/pypi/requests/2.21.0/json and nothing else. But it brings an extra burden to maintain that service. IMO it would be more ideal if it is run by PyPI, or PyPI should fix the unreliability of the JSON API.

If that is the case PDM also caches the dependency info in local files. It will be slow for the first run and consulting the local caches afterward. The slowness of the dependency solving could also be caused by potential loops in the resolution. You can observe it by adding -v to the command. Paste the log if you find any suspicious loops.

What's more, due to the unreliability of the PyPI JSON API, PDM disables it by default, but you can turn it on by pdm config pypi.json_api on.