Poetry is extremely slow when resolving the dependencies

[qiuwei]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Centos 7
• Poetry version: 1.0.0
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/qiuwei/a0c7eee89e5e8d75edb477858213c30b

Issue

I created an empty project and run poetry add allennlp. It takes ages to resolve the dependencies.

[David-OConnor]

Could this be due to downloading packages from pypi to inspect their dependencies, when not properly specified?

[qiuwei]

Could this be due to downloading packages from pypi to inspect their dependencies, when not properly specified?

It seems so. I have checked the detailed log, poetry kept retrying to resolve the dependency for botocore, but without success. So I assume that the dependency can be eventually resolved if enough time is given.

However, is there any way to get around this?

BTW, I also consider it's better to give some warning if there are some dependencies are not properly specified and could not be resolved after a number of attempts.

[thmsklngr]

Hi,

I encounter a similiar problem on my MacOS. Python version used is 3.7.6, Poetry is 1.0.5. I just created a new project with no dependencies so far in pyproject.toml, just initially pytest. It takes ages until the new virtualenv is setup with all 11 packages installed.

Running it with -vvv does not bring any new findings.

Regards, Thomas

[yalunren]

Yes, i'm running into the same problem. Resolving dependencies takes forever. I tried to use VPN to get through the GFW, nevertheless, it is still not working. I also tried to change pip source and wrote local source in the toml file, neither works. It's driving me nuts.

[long2ice]

same here...😱

[luanfrv9]

Same here. I just created an empty project then ran poetry install and it takes so much time to resolve dependencies.

[luanfrv9]

I'm currently using this workaround:

poetry export -f requirements.txt > requirements.txt
python -m pip install -r requirements.txt
poetry install

It takes a lower time space to install the package locally since all deps are already installed.

Make sure to run poetry shell before to access the created virtual environment and install on it instead of on user/global path.

[MartinThoma]

Poetry being slow to resolve dependencies seems to be a reoccuring issue:

• Poetry resolving dependencies is amazingly slow #476 - Poetry resolving dependencies is amazingly slow
• Resolving dependencies are slow #819 - Resolving dependencies are slow
• Poetry update never finishes resolve and Poetry show --outdated hangs #832 - Poetry update never finishes resolve and Poetry show --outdated hangs
• Poetry doesn't resolve on MacOs Mojave #1047 - Poetry doesn't resolve on MacOs Mojave

[aoreki]

Maybe there is a dependency conflict.

[gagarine]

No conflict. Poetry is slow as hell.

[sdispater]

First of all, I want to say there is ongoing work to improve the dependency resolution.

However, there is so much Poetry can do with the current state of the Python ecosystem. I invite you to read https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow to know a little more about why the dependency resolution can be slow.

If you report that Poetry is slow, we would appreciate a pyproject.toml that reproduces the issue so we can debug what's going on and if it's on Poetry's end or just the expected behavior.

@gagarine Could you provide the pyproject.toml file you are using?

[gagarine]

Take about 2min to resolve dependencies after adding newspaper3k on a fresh project.
Connexion: 40ms ping and 10Mb/s down.

pyproject.toml

[tool.poetry]
name = "datafox"
version = "0.1.0"
description = ""
authors = ["Me <mail@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.8"
newspaper3k = "^0.2.8"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[David-OConnor]

Hey dudes - as Sebastian implied, the root cause is the Python eco's inconsistent/incomplete way of specifying dependencies and package metadata. Unfortunately, the Pypi team is treating this as a wont fix.

In particular, using the Pypi json endpoint, an empty dep list could either mean "no dependencies", or "dependencies not specified". The Pypi team doesn't want to differentiate between these two cases for reasoning I don't follow.

The soln is to workaround this by maintaining a sep cache from Pypi that properly handles this distinction, and perhaps refuse to use packages that don't properly specify deps. However, this latter aspect may be tough, due to long dep nests.

Python's grown a lot over the decades, and it has much remaining from its early days. There's a culture of no-breaking-changes at any cost.

Having to run arbitrary python code to find dependencies is fucked, but .... we can do this for each noncompliant package, and save it.

[pradyunsg]

First, it's capitalized PyPI.

Second, there is no way for PyPI to know dependencies for all packages without executing arbitrary code -- which is difficult to do safely and expensive (computationally and financially). PyPI is run on donated infrastructure from sponsors, maintained by volunteers and does not have millions of dollars of funding like many other language ecosystems' package indexes.

---

For anyone interested in further reading, here's an article written by a PyPI admin on this topic: https://dustingram.com/articles/2018/03/05/why-pypi-doesnt-know-dependencies/

[David-OConnor]

It's not as tough as you imply.

You accept some risk by running the arbitrary code, but accepting things as they are isn't the right approach. We're already forcing this on anyone who installs Python packages; it's what triggers the delays cited in this thread.

I have the above repo running on a $10/month Heroku plan, and it works well.

I've made the assumption that if dependencies are specified, they're specified correctly, so only check the ones that show as having no deps. This won't work every time, but does in a large majority of cases.

Related: Projects like Poetry are already taking a swing at preventing this in the future: Specifying deps in pyproject.toml, Pipfile etc.

[pradyunsg]

A personal Heroku app is not going to be as valuable a target as PyPI would be. Neither is a $10/month Heroku app going to be able to support the millions of API requests that PyPI gets everyday. The problem isn't in writing a script run a setup.py file in a sandbox, but in the logistics and challenges of providing it for the entire ecosystem.

"It works 90% of the time" is not an approach that can be taken by the canonical package index (which has to be used by everyone) but can be taken by specific tools (which users opt into using). Similar to how poetry can use an AST parser for setup.py files which works >90% of the time, to avoid the overhead of a subprocess call, but pip shouldn't.

Anyway, I wanted to call out that "just blame PyPI folks because they don't care/are lazy" is straight up wrong IMO -- there are reasons that things are the way they are. That doesn't mean we shouldn't improve them, but it's important to understand why we're where we are. I'm going to step away now.

[David-OConnor]

Before you step away - Can you think of a reason PyPi shouldn't differentiate between no dependencies, and missing dependency data?

If going through existing releases is too bold, what about for new ones?

[gagarine]

I'm new to (more serious) python and don't understand the big drama. Yet setup.py seems a powerful and very bad idea. Dependency management is terrible in python, because of setup.py?

Can someone post a couple of examples where a txt file is not enough andsetup.py was absolutely necessary?

"is a feature that has enabled better compatibility across an increasingly broad spectrum of platforms."

Cargo do it like this: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#platform-specific-dependencies this is not enough for python?

Why poetry does not create their own package repository, avoiding setup.py and using their own dependency declaration? Could take time... but a bot can automatise the pull request on most python module based on the kind of technics used in https://github.com/David-OConnor/pydeps

[David-OConnor]

I think the root cause is Python's been around for a while, and tries to maintain backwards compatibility. I agree - setup.py isn't an elegant way to do things, and a file that declares dependencies and metadata is a better system. The wheel format causes dependencies to be specified in a MANIFEST file, but there are still many older packages that don't use this format.

As a new lang, Rust benefited by learning from the successes and failures of existing ones. Ie it has nice tools like Cargo, docs, clippy, fmt etc. It's possible to to implement tools / defaults like this for Python, but involves a big change, and potentially backwards-incompatibility. There are equivalents for many of these (pyproject.toml, black etc), but they're not officially supported or widely-adopted. Look at how long it took the Python 3 to be widely adopted for a taste of the challenge.

[MartinThoma]

Can someone post a couple of examples where a txt file is not enough andsetup.py was absolutely necessary?

Not absolutely necessary, but helpful in the following scenario:

• A package has extra A and B
• Extra B needs extra A

With setup.py, you can follow the DRY principle:

requires_a = ('some', 'thing')
requires_b = requires_a + ('foo', 'bar')

For requirements.txt, I'm on the one hand not sure how you denote extras at all and even if you can, you would need to repeat the requirements of a within the requirements of b. This is prone to human error.

However, while creating the package, the package builder could output a textfile having those requirements.

[MartinThoma]

Why poetry does not create their own package repository

You mean replacing PyPI? Good luck with that. I analyzed the packages on PyPI in January (PyPI Analysis 2020):

• 208,492 packages in total
• 2,957 had a pyproject.toml
• 1,511 specified poetry as a tool

I also gave a course about packaging in Python this year to PhD students. They simply want to share there work to a broad audience. I only mentioned poetry briefly because it is such a niche right now.

Changing a big, working system is hard. It took Python 2 -> 3 about 12 years and it is still not completely finished.

[finswimmer]

Hi,

I would like to invite everone interested in how depedencies should be declared to this discussion on python.org

fin swimmer

[gagarine]

@finswimmer I check the discussion. Seem like they are reinventing the wheel instead of copy/past something that works (composer, Cargo, ...).

For requirements.txt, I'm on the one hand not sure how you denote extras at all and even if you can, you would need to repeat the requirements of a within the requirements of b. This is prone to human error.

For sure requirements.txt is not good.

You mean replacing PyPI? Good luck with that.

Yes. But why making poetry if it's not to replace PyPI and requirements.txt?

If poetry is compatible with PyPI, there is no incentive to add a pyproject.toml. Perhaps I don't even know I should add one. Now if every time I try to install a package that has no pyproject.toml the command line proposes me to open an issue on this project with a ready to use a template, this could speed things up.

[pradyunsg]

Can you think of a reason PyPi shouldn't differentiate between no dependencies, and missing dependency data?

It'd be more productive to file an issue on https://github.com/pypa/warehouse, to ask this. There's either a good reason, or PyPI would be open to adding this functionality. In the latter case, depending on how the details work out, it might need to be standardized like pyproject.toml was before poetry adopted it, so that the entire ecosystem can depend on and utilize it.

[japita-se]

My two cents solution.
I mistakenly add pytest in the dependencies. I deleted from pyproject.toml and installed in the virtaulenv:

poetry shell
pip install pytest

And now it is fine.
You have to find which module/package is blocking the resolver. Sometimes, you can install the binary and not the module.

[nikita-davydov]

For me, the issue was fixed by setting particular version of package that I want to install, not poetry add "package_name<2", but poetry add "package_name==1.18"

[tall-josh]

TLDR: uninstalling 1.1.x and installing 1.2 worked for me.

I also tried poetry adding each dep individually and got an error:

that was resolved with python -m pip install --upgrade pip.

---

After that I noticed poetry add "pandas@^1.0" was hanging so I put started a new venv and left pandas 'till the end. Sure enough everything added just fine until it got to pandas. Not sure what's up with pandas. That's when I rm -rfed all poetry related stuff and installed poetry 1.2

[Kache]

Running a poetry add s3path -vvv (/w 1.1.14) and resolving just one particular constraint has taken over 20 mins (and is still going), starting with:

   1: fact: no versions of boto3 match >1.16.35,<1.16.36 || >1.16.36,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42>
   1: conflict: no versions of boto3 match >1.16.35,<1.16.36 || >1.16.36,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.1>
   1: !  boto3 (>1.16.35,<1.16.36 || >1.16.36,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<>
   1: ! which is caused by "boto3 (1.16.35) depends on botocore (>=1.19.35,<1.20.0)"
   1: ! thus: boto3 (>=1.16.35,<1.16.36 || >1.16.36,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.1>
   1: !  boto3 (>=1.16.35,<1.16.36 || >1.16.36,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,>
   1: ! which is caused by "boto3 (1.16.36) depends on botocore (>=1.19.36,<1.20.0)"
   1: ! thus: boto3 (>=1.16.35,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.1>
   1: !  boto3 (>=1.16.35,<1.16.37 || >1.16.37,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,>
   1: ! which is caused by "boto3 (1.16.37) depends on botocore (>=1.19.37,<1.20.0)"
   1: ! thus: boto3 (>=1.16.35,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.1>
   1: !  boto3 (>=1.16.35,<1.16.38 || >1.16.38,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.16.45,>
   1: ! which is caused by "boto3 (1.16.38) depends on botocore (>=1.19.38,<1.20.0)"
   1: ! thus: boto3 (>=1.16.35,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.16.45,<1.16.46 || >1.1>
   1: !  boto3 (>=1.16.35,<1.16.39 || >1.16.39,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.16.45,<1.16.46 || >1.16.46,>
   1: ! which is caused by "boto3 (1.16.39) depends on botocore (>=1.19.39,<1.20.0)"
   1: ! thus: boto3 (>=1.16.35,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.16.45,<1.16.46 || >1.16.46,<1.16.47 || >1.1>
   1: !  boto3 (>=1.16.35,<1.16.40 || >1.16.40,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.16.45,<1.16.46 || >1.16.46,<1.16.47 || >1.16.47,>
   1: ! which is caused by "boto3 (1.16.40) depends on botocore (>=1.19.40,<1.20.0)"
   1: ! thus: boto3 (>=1.16.35,<1.16.41 || >1.16.41,<1.16.42 || >1.16.42,<1.16.43 || >1.16.43,<1.16.44 || >1.16.44,<1.16.45 || >1.16.45,<1.16.46 || >1.16.46,<1.16.47 || >1.16.47,<1.16.48 || >1.1>

The first line of constraints is nearly 9000 chars long (the above is truncated), each iteration appears to output one less constraint at a time, and each iteration takes 3~6 sec.

What the heck?

edit

It's speeding up as the list gets shorter (this is some O(n^2) business...), and it ends with the following:

   1: !  boto3 (>=1.16.35,<1.24.68 || >1.24.68,<1.24.69 || >1.24.69,<2) is partially satisfied by not  boto3 (1.24.68)
   1: ! which is caused by "boto3 (1.24.68) depends on botocore (>=1.27.68,<1.28.0)"
   1: ! thus: boto3 (>=1.16.35,<1.24.69 || >1.24.69,<2) requires botocore (>=1.19.35,<1.28.0)
   1: fact: boto3 (>=1.16.35,<1.24.69 || >1.24.69,<2) requires botocore (>=1.19.35,<1.28.0)
   1: derived: not boto3 (>=1.16.35,!=1.24.69,<2)
   1: derived: s3transfer (>=0.6.0,<0.7.0)
   1: conflict: boto3 (1.24.69) depends on botocore (>=1.27.69,<1.28.0)
   1: !  boto3 (1.24.69) is partially satisfied by not  boto3 (>=1.16.35,<1.24.69 || >1.24.69,<2)
   1: ! which is caused by "boto3 (>=1.16.35,<1.24.69 || >1.24.69,<2) requires botocore (>=1.19.35,<1.28.0)"
   1: ! thus: boto3 (>=1.16.35,<2) requires botocore (>=1.19.35,<1.28.0)
   1: fact: boto3 (>=1.16.35,<2) requires botocore (>=1.19.35,<1.28.0)
   1: derived: botocore (>=1.19.35,<1.28.0)
<REDACTED>: 418 packages found for botocore >=1.19.35,<1.28.0

edit2

Now it's repeating all of the above while still within the same run? Seems like there's a serious amount of backtracking going on.

Output of `rg -C2 'solving took' poetry_add_s3path.log`:

   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 73.032 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 52.206 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 55.441 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 56.791 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 60.024 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.1', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   2: fact: jpype1 (1.3.0) depends on typing_extensions (*)
   2: selecting jpype1 (1.3.0)
   2: Version solving took 3350.503 seconds.
   2: Tried 2 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.1', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}}).
--
   2: fact: jpype1 (1.3.0) depends on typing_extensions (*)
   2: selecting jpype1 (1.3.0)
   2: Version solving took 2841.885 seconds.
   2: Tried 2 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 58.249 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}, Package('sentry-sdk', '1.9.1', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   2: fact: jpype1 (1.3.0) depends on typing_extensions (*)
   2: selecting jpype1 (1.3.0)
   2: Version solving took 2699.238 seconds.
   2: Tried 2 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}, Package('sentry-sdk', '1.9.1', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}}).
--
   2: fact: jpype1 (1.3.0) depends on typing_extensions (*)
   2: selecting jpype1 (1.3.0)
   2: Version solving took 2552.779 seconds.
   2: Tried 2 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 48.207 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).
--
   0: Duplicate dependencies for urllib3
   0: Different requirements found for urllib3 (>=1.26.9) with markers python_version >= "3.5" and urllib3 (>=1.26.11) with markers python_version >= "3.6".
   1: Version solving took 49.597 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('sentry-sdk', '1.9.5', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.4', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.3', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.11)>}, Package('sentry-sdk', '1.9.2', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}, Package('sentry-sdk', '1.9.1', features=frozenset({'flask'}), source_type='legacy', source_url='https://unata.jfrog.io/unata/api/pypi/redacted-pypi-virtual/simple', source_reference='redacted'): {'urllib3': <Dependency urllib3 (>=1.26.9)>}}).

[dimbleby]

There are cases where poetry during its search first chooses foo, which turns out to be incompatible with all versions of bar, but there are hundreds or even thousands of versions of bar. (The AWS ecosystem seems to like to do this).

Then poetry has to search through all of those versions of bar before it can finally conclude that there's a problem and try the next version of foo. I think there is indeed something quadratic going on here.

when you encounter such cases the pragmatic thing to do is to put an explicit constraint on bar (here boto3) forcing it to be recent. Then poetry can find conflicts fast.

[Kache]

@dimbleby That doesn't seem to always work:

 slack-sdk = "^3.17.2"
+boto3 = "1.17.49"
+ddtrace = "0.34.2"

Yet it still appears to search through unrelated ddtrace versions:

   1: fact: environ-config (21.2.0) depends on attrs (>=17.4.0)
   1: selecting environ-config (21.2.0)
   1: selecting python-dotenv (0.19.2)
   1: fact: ddtrace (1.4.4) depends on ddtrace (1.4.4)
   1: fact: ddtrace (1.4.4) depends on ddsketch (>=2.0.1)
   1: fact: ddtrace (1.4.4) depends on packaging (>=17.1)
   1: fact: ddtrace (1.4.4) depends on tenacity (>=5)
   1: fact: ddtrace (1.4.4) depends on attrs (>=19.2.0)
   1: fact: ddtrace (1.4.4) depends on six (>=1.12.0)
   1: fact: ddtrace (1.4.4) depends on typing_extensions (*)
   1: fact: ddtrace (1.4.4) depends on importlib_metadata (*)
   1: fact: ddtrace (1.4.4) depends on bytecode (>=0.13.0,<0.14.0)
   1: fact: ddtrace (1.4.4) depends on protobuf (>=3)
   1: fact: ddtrace (1.4.4) depends on opentracing (>=2.0.0)
   1: derived: not ddtrace[opentracing] (==1.4.4)
   1: fact: ddtrace (1.4.3) depends on ddtrace (1.4.3)
   1: fact: ddtrace (1.4.3) depends on ddsketch (>=2.0.1)
   1: fact: ddtrace (1.4.3) depends on packaging (>=17.1)
   1: fact: ddtrace (1.4.3) depends on tenacity (>=5)
   1: fact: ddtrace (1.4.3) depends on attrs (>=19.2.0)
   1: fact: ddtrace (1.4.3) depends on six (>=1.12.0)
   1: fact: ddtrace (1.4.3) depends on typing_extensions (*)
   1: fact: ddtrace (1.4.3) depends on importlib_metadata (*)
   1: fact: ddtrace (1.4.3) depends on bytecode (>=0.13.0,<0.14.0)
   1: fact: ddtrace (1.4.3) depends on protobuf (>=3)
   1: fact: ddtrace (1.4.3) depends on opentracing (>=2.0.0)
   1: derived: not ddtrace[opentracing] (==1.4.3)
   1: fact: ddtrace (1.4.2) depends on ddtrace (1.4.2)
   1: fact: ddtrace (1.4.2) depends on ddsketch (>=2.0.1)
   # this continues on...

Any suggestions?

[dswalter]

@Kache, It appears to search through dependencies depth-first, rather than breadth-first. As a result, you've probably got a something earlier in your pyproject.toml that depends on ddtrace, so the dependency resolver grabbed that version and tried to resolve using that, rather than the ddtrace version you've specified.

I've had some success moving the dependencies I want exact version logic prioritizing earlier in the pyproject.toml file.

(I also disabled IPv6, upgraded to poetry 1.2x, and have reduced the possible space for the troubling aws libraries (boto3 and awscli, for me) so those go at the very end of my dependency file and have only a few recent versions to chew through.

I'm seeing dependency resolution time between 5 and 35 seconds most of the time now.

[justinTM]

i think if you lose internet connectivity during a download of a dependency, it seems to introduce the possibility of making subsequent retry operations hang.

i did a pip install and my machine went to sleep during the download of pyspark (200MB or something). when i woke it up, it had been "resolving dependencies" for 10k seconds. Ctrl-C and a retry of the command led to the same issue.

Running the above workaround resulted in a similar hang, but this time on the lock file:

sh-3.2$ poetry export -f requirements.txt > requirements.txt
Configuration file exists at /Users/me/Library/Application Support/pypoetry, reusing this directory.

Consider moving configuration to /Users/me/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.
The lock file does not exist. Locking.

it's been locking for 10-20 minutes now.

hope this helps at all

[justinTM]

this fixed my issue:
poetry cache clear --all pypi

[tall-josh]

Re:

this fixed my issue: poetry cache clear --all pypi

I've seen this mentioned a few times, if it works that's great, but why would it work? Wouldn't clearing the cache cause things to take longer? What am I missing?

[johnthagen]

@tall-josh Because Poetry had (maybe still has?) a bug where corrupted cache entries make Poetry hang forever when trying to resolve dependencies.

For me this seemed to occur if I Ctrl+C'd Poetry while it was doing an install and it was downloading packages.

I observed this on 1.1.14, so perhaps it's fixed in 1.2+.

• Lockfile generation hangs indefinitely when cache is corrupted #5922

[neersighted]

Clearing the cache is most likely related to clearing out partial/incomplete/corrupted by concurrent usage downloads that can cause an indefinite hang.

[namekkozZ]

add this to .toml file is work for me
[[tool.poetry.source]] name = 'aliyun.mirrors' url = 'https://mirrors.aliyun.com/pypi/simple/' default = true

[legioz]

Still have same problem on version 1.2.1

[neersighted]

Hi all,

This issue has gotten quite long and meandering, with many disparate causes, solutions, fixed issues, perhaps still extant bugs, and many "me too" comments all discussed.

I'm going to close this issue as most of the root causes discussed within have either been solved in 1.2 or 1.3 (the changes in 1.3 are behavior changes not eligible for backport).

If you are having issues with Poetry taking a long time to resolve dependencies, please first open a Discussion or start on Discord, as many of them are related to configuration and large search space (basically, you're creating exponential work for the solver and should tighten your constraints). Tooling to advise the user should be possible (if difficult to develop) in the long run, and anyone interested in tackling this hairy problem should reach out to the team via a Discussion or on Discord.

Past that, please make sure to test with the latest code (both on the 1.2 branch and master branch presently) when trying to reproduce resolver issues as we are making improvements all the time, and your issue may be fixed and pending release already.

Finally, good reproductions are needed for this category of issue. Many times they are related to transient network issues, pathologically bad cases due to decisions made around (low traffic, private) custom package indexes, or a corrupted cache/bad local config. Reproducing in a container with publicly available packages will mean that someone can dissect your issue and possibly fix it. If you can't reproduce it with public packages, but you can with private packages, there are still options -- everything from sharing details with the team in private, to creating 'imitation' repositories to reproduce an issue.

Please refrain from commenting on this issue more if it's not a mitigation/common solution -- "me too" is not very helpful and will send dozens of emails, and if you can reproduce bad performance consistently/in a clean environment it should be an issue. If you're stuck and need help, ask for support using one of the methods mentioned above.

[legioz]

@Kache, It appears to search through dependencies depth-first, rather than breadth-first. As a result, you've probably got a something earlier in your pyproject.toml that depends on ddtrace, so the dependency resolver grabbed that version and tried to resolve using that, rather than the ddtrace version you've specified.

I've had some success moving the dependencies I want exact version logic prioritizing earlier in the pyproject.toml file.

(I also disabled IPv6, upgraded to poetry 1.2x, and have reduced the possible space for the troubling aws libraries (boto3 and awscli, for me) so those go at the very end of my dependency file and have only a few recent versions to chew through.

I'm seeing dependency resolution time between 5 and 35 seconds most of the time now.

I have a private repository from gitlab which is know to have some problems with ipv6, I disabled ipv6 too and add this to my .ssh/config

Host gitlab.com
    AddressFamily inet

then sudo service ssh restart

Got a huge improvement in dependencies resolver

[ADR-007]

I found one workaround:
You can specify requirements more precisely. E.g.: not python="^3.10" but python="^3.10.0"

[neersighted]

Reducing the search space is not a workaround, but a common issue (as mentioned in my comment above). That being said, the version you have written should be equivalent (^3.10.0 and ^3.10) -- any differences you noticed with that change are likely unrelated.

[ADR-007]

Actually, they a not equivalent: ^3.10 means >=3.10,<4 but ^3.10.0 means >=3.10,<3.11
It reduces time from >2000s (I didn't wait more) to 200s

[neersighted]

That should expand to python_full_version >= "3.10.0" and python_version < "4.0" -- ^ is not defined to operate differently based on precision as far as I am aware. However, we are generating a lot of traffic on a tangential topic -- let's take it to a Discussion or Discord if you think my explanation is insufficient or incorrect.