Poetry install is stuck on specific dependency with "pending..."

[cpvandehey]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: 10.15.7
• Poetry version: 1.1.4, 1.1.3, 1.0.9

Issue

Hello poetry crew. Loving the work that yall do!

I am having issues with poetry while running poetry install in one of my projects on one specific machine.

It appears to get stuck on one dependency during the install and it will hang forever. When I kill the process, I see this in my shell (hinting at a race condtion/locking problem):

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/Users/{}/.pyenv/versions/3.6.8/lib/python3.6/concurrent/futures/thread.py", line 40, in _python_exit
    t.join()
  File "/Users/{}/.pyenv/versions/3.6.8/lib/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/Users/{}/.pyenv/versions/3.6.8/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

I am on the latest poetry (1.1.4) and have tried 1.1.3 as well with the same results (all of which while running python 3.6.8 (I did use 3.6.5 as well)).

After downgrade to 1.0.9, I no longer faced this issue. I am assuming that there was an update to the way poetry can download multiple deps at the same time. Anyone else facing locking issues? At the very least, poetry should recognize a pending download with no progress being made to exit and fail early, but it simply hung and would never exit (left it going for at least 30 mins).

[sinoroc]

Maybe it would be helpful to see the pyproject.toml. Is that on macos? Were you able to identify if it hangs always on the same dependency or not?

[cpvandehey]

Unfortunately, I cannot share the pyproject.toml as it contains a lot of proprietary names.

Is that on macos?

As noted above, mac OS 10.15.7 (catalina)

Were you able to identify if it hangs always on the same dependency or not?

It does indeed hang on the same dependency every time.

---

One other interesting thing to note:
I can manually go to the venv created by poetry and go to the pip that lives in its bin directory. When I pip install the problematic dependency (that hangs in the poetry install execution), it does install correctly. So it appears to focus the problem even further on the locking mechanism being used.

[DeepDarkOdyssey]

Same issue here. I just upgrade from v1.0.0, but got pending on some packages during installing , typically numpy , sklearn... The error message shows it got a read time out from my nexus repo, but it never happened before. And after I try the poetry install several times, it installed all package successfully. Does the parallel installation trigger this issue?

[sinoroc]

I do not know how to help you all here...

My advice would be to try one of those (maybe in that order) if you have not yet:

• delete the poetry.lock file and retry
• delete the virtual environment and retry
• deactivate the new (parallel) installer with poetry config experimental.new-installer false and retry (do not forget to reactivate it afterwards if it proves to have no influence)
• or any combination of those

Also maybe provide some verbose output (-vvv), that might be helpful for when one of the maintainers gets the time to handle this ticket. Of course, the best as always is if you can provide a minimal reproducible example that triggers the failure.

[arnegroskurth]

I do experience the same problem with poetry 1.1.4 within a python 3.9 docker container. The error goes away by following the suggestion of @sinoroc to deactivate the experimental installer.

pyproject.toml:

[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.8"
watchdog = "^1.0.1"

[tool.poetry.dev-dependencies]
docker = "^4.4.0"
pytest = "^6.2.1"

poetry.lock.txt

[vaidik]

I can confirm that disabling the experimental parallel installer worked. My environment, although, was Docker, particularly the python:3.8-slim image.

[ohthehugemanatee]

Same problem here, and disabling the experimental installer fixed it for me, too. My environment is this docker image: mcr.microsoft.com/vscode/devcontainers/python:0-3.8 with the following commands:

apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends python3-venv gnupg2
pip install tox
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
source $HOME/.poetry/env
poetry --version
poetry install -v

[jlucas91]

Could someone on the Poetry team provide a perspective on prioritization for this? It seems like there's meaningful evidence that this is an issue independent of environment or dependencies. Turning off the installer is a good workaround in the interim, but it significantly increases install times. I'd love to know I have some hope of being able to turn it back on at some point.

[dbowring]

Disabling experimental installer also fixes this for me, but also able to fix it using the experimental installer by adding a timeout for the requests to pypi, E.g.

diff --git a/poetry/repositories/pypi_repository.py b/poetry/repositories/pypi_repository.py
index 1c7b7be3..a1f0922a 100644
--- a/poetry/repositories/pypi_repository.py
+++ b/poetry/repositories/pypi_repository.py
@@ -321,14 +321,14 @@ class PyPiRepository(RemoteRepository):
 
         return data.asdict()
 
-    def _get(self, endpoint: str) -> Union[dict, None]:
+    def _get(self, endpoint: str, timeout: int = 5) -> Union[dict, None]:
         try:
-            json_response = self.session.get(self._base_url + endpoint)
+            json_response = self.session.get(self._base_url + endpoint, timeout=timeout)
         except requests.exceptions.TooManyRedirects:
             # Cache control redirect loop.
             # We try to remove the cache and try again
             self._cache_control_cache.delete(self._base_url + endpoint)
-            json_response = self.session.get(self._base_url + endpoint)
+            json_response = self.session.get(self._base_url + endpoint, timeout=timeout)
 
         if json_response.status_code == 404:
             return None

[GerevSec]

Happens on WSL2 ubuntu 20.04 with python3.10

[schlich]

Recent issues possibly related to discussion happening in #2094

[epomatti]

On Ubuntu 20.04 WSL2 here. Only disabling IPv6 did the trick:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1

[dimbleby]

#3352 (comment) suggests a very plausible fix, perhaps someone would like to submit an MR.

[andreas-vester]

The error is still prevalent in v1.2.0.

Deactivating the experimental installer solves the issue for now. However, a more thorough solution would be appreciated.

[dimbleby]

so far as I can see this thread contains no way to reproduce the problem. Please provide a repro, if you have one.

[neersighted]

I'm going to close this as it is years old and crosses over many versions (and the old and new installers)! For anyone coming across this, if you can reproduce this in a consistent way, please open an issue for the specific case you are having trouble with.

[jleclanche]

@neersighted timeouts were added in 584aaee via #5881, they were mentioned above as a potential (or at the very least they are probably hiding the underlying issue now, and it's just slower than it should be rather than stuck).

[neersighted]

There's multiple underlying causes both theorized and fixed in this issue -- that's why I'd like anybody who can consistently reproduce this behavior to open a new issue so we can reason about any specific instance or fix in one place.

[cah-zack-strife]

I think I can recreate this issue. If a user is waiting for ssl certificates too long the lock seems to get enabled. After that it will hang forever waiting for unlocks. Turning off the parallel will fix it either way but maybe this is a way to help recreate the issue.

[zachliu]

it also happens under:

pip 22.2.2
poetry 1.2.1
python 3.8.5 and also 3.10.0

traceback:

  • Installing cachetools (5.0.0): Pending...
  • Installing jmespath (0.10.0): Pending...
  • Installing oauthlib (3.2.0): Pending...
  • Installing pyasn1-modules (0.2.8): Pending...
  • Installing python-dateutil (2.8.2): Pending...
  • Updating requests (2.28.1 -> 2.27.1): Pending...
  • Installing rsa (4.8): Pending...
  • Installing smmap (5.0.0): Pending...
^C^C^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/zach/.asdf/installs/python/3.8.5/lib/python3.8/concurrent/futures/thread.py", line 40, in _python_exit
    t.join()
  File "/home/zach/.asdf/installs/python/3.8.5/lib/python3.8/threading.py", line 1011, in join
    self._wait_for_tstate_lock()
  File "/home/zach/.asdf/installs/python/3.8.5/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

workaround:

poetry config experimental.new-installer false

it started to happen this week, last week the same command(s) worked fine 🤔

[neersighted]

If you can reproduce this reliably in a container or clean environment, please open a new issue with the reproduction. There are many possible factors from cache corruption, network issues, etc that could influence this, so a clean reproduction is key.

[Purg]

FWIW I deleted my poetry cache and artifacts (~/.cache/pypoetry/cache/), and found "things" to work again. If there is any wisdom as to when/why corruption in these locations leads to such an inability to install venvs, that would be educational.

[zachliu]

@Purg thank you sire! i've been stuck on this for hours... i tried all these:

1. turn off the experimental.new-installer

   poetry config experimental.new-installer false
   

   but this only worked "for a moment" then stopped working:

   

2. delete virtualenv - didn't work

3. delete poetry.lock - didn't work

4. clear all cache by running (see Poetry is extremely slow when resolving the dependencies #2094 (comment))

   $ poetry cache list
   $ poetry cache clear --all pypi 
   

   didn't work

5. only yours worked:

   cd ~/.cache/pypoetry/cache/
   rm -rf artifacts/ cache/
   

[neersighted]

The reason for corrupted caches is either:

1. cancelled downloads leaving a orphan lock file (fixed by Use filelock to lock the file cache #6471)
2. multiple concurrent instances of Poetry writing to the same file (fixed by Fix a race condition in simultaneous poetry installations updating the cache #6186)

Both of these will not be backported to 1.2 as they are new changes to Poetry behavior/not fixes for regressions, but they will be available in the near future, and you can test them today by installing from master. Clearing the cache is a standard troubleshooting step, and always worth doing (indeed --no-cache is a handy global flag that lets you ignore the cache without actually clearing it).

[zachliu]

my last question: what's the difference between

$ poetry cache list
$ poetry cache clear --all pypi 

and

cd ~/.cache/pypoetry/cache/
rm -rf artifacts/ cache/

?

[neersighted]

poetry cache clear is like deleting the cache folder, but does not affect the artifacts folder.

[zachliu]

artifacts are just whl packages, maybe they are corrupted too? 🤔
i'll remember to clear both artifacts and cache in the future

[neersighted]

Artifacts are distfiles (wheels, sdists) and can be corrupted by an interrupted Poetry (ctrl-c, machine shutdown) as well as a concurrent Poetry. This will be fixed in the 1.3 release.

[TallGirlVanessa]

I had problems with this due to "Connection pool is full, discarding connection: pypi.org. Connection pool size: 10". It helped to set poetry config installer.max-workers 10. Should we consider that as an upper bound to the default? I think I might have something like 8 cores, which takes the worker number to 12.

[rberger]

The artifacts and cache dirs were in ~/Library/Caches/pypoetry on my M1 Mac for those who can't find it in ~/.cache/pypoetry/cache/

cd ~/Library/Caches/pypoetry
rm -rf artifacts/ cache/

Doing that cleared things up for me. Before that, it was consistently hanging on trying to fetch flake8-bugbear from pypy

[neersighted]

You can always find the cache directory with poetry config cache-dir. If you're having issues you think might be related, try clearing the cache first. If you're still having issues, I'd suggest opening a Discussion or joining Discord first as 90% of the time these are local config issues.

I would ask that if you don't have anything to add that hasn't already been said in this issue, you hold off, as generating notifications for 20+ people for a root cause that is solved is unnecessary (and if you have something novel, we want to identify that and make sure it's tracked as a new issue).

[bhuiyanmobasshir94]

• poetry config experimental.new-installer false

This solution helped me. Thank you.

[Lamarcke]

I'm facing this issue in the latest Poetry version.
This helps:

poetry config experimental.new-installer false

[PitterRup]

I do not know how to help you all here...

My advice would be to try one of those (maybe in that order) if you have not yet:

* delete the `poetry.lock` file and retry

* delete the virtual environment and retry

* deactivate the new (parallel) installer with `poetry config experimental.new-installer false` and retry (do not forget to reactivate it afterwards if it proves to have no influence)

* or any combination of those

Also maybe provide some verbose output (-vvv), that might be helpful for when one of the maintainers gets the time to handle this ticket. Of course, the best as always is if you can provide a minimal reproducible example that triggers the failure.

In my case deactivating the new (parallel) installer solved the problem. It was stuck on installing pycparser 2.21 and six 1.16.0 (maybe it will be useful)

[neersighted]

If deactivating the new installer helps, you have a corrupted artifact cache. Please rm -rf $(poetry config cache-dir)/artifacts and try again. I am locking this issue as it keeps misleading people -- most of the issues here are solved, including the artifact corruption issue. Disabling the new installer is not sustainable (as the old installer is scheduled for removal), so please follow the troubleshooting steps outlined.

If, after doing the following, you still reproduce on the latest version of Poetry, please open a new issue so we can fix bugs in the new installer:

• poetry env remove --all
• poetry cache clear --all .
• rm -rf $(poetry config cache-dir)/artifacts