Poetry resolving dependencies is amazingly slow

[cprieto]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 10.14
• Poetry version: 0.11.5
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/cprieto/b5996c4ffd2e77c7ab05253022c482bd

Issue

I just created a new project with poetry new and just added one package, petl, I was surprised when I found out in a new project (which includes pytest) it took over half an hour to install, yes, I know no cache was used, and yes, I know it has to download all the dependencies and handle their installation, but this time was insane.

I know this has been reported before in #65 but maybe I am missing some discussion about how to improve this time, because honestly, waiting between 3min to 30min to install a package is too much, especially when the package size is small.

I included the output of my time poetry add -vvv petl command: https://gist.github.com/cprieto/ed5def2f51e8e16328770ebd76a6355e

[sdispater]

There must be something wrong on your system.

Here is my output on a fresh cache (both pip and poetry caches):

Creating virtualenv sample-py3.7 in /Users/sebastien/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/sebastien/Library/Caches/pypoetry/virtualenvs/sample-py3.7
PyPI: 61 packages found for petl *
Using version ^1.2 for petl

Updating dependencies
Resolving dependencies...
   1: fact: sample is 0.1.0
   1: derived: sample
   1: fact: sample depends on petl (^1.2)
   1: fact: sample depends on pytest (^3.0)
   1: selecting sample (0.1.0)
   1: derived: pytest (^3.0)
   1: derived: petl (^1.2)
PyPI: 39 packages found for pytest >=3.0,<4.0
PyPI: 1 packages found for petl >=1.2,<2.0
PyPI: Getting info for petl (1.2.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: petl-1.2.0.tar.gz
   1: selecting petl (1.2.0)
PyPI: Getting info for pytest (3.8.2) from PyPI
   1: fact: pytest (3.8.2) depends on colorama (*)
   1: fact: pytest (3.8.2) depends on pathlib2 (>=2.2.0)
   1: fact: pytest (3.8.2) depends on funcsigs (*)
   1: fact: pytest (3.8.2) depends on pluggy (>=0.7)
   1: fact: pytest (3.8.2) depends on atomicwrites (>=1.0)
   1: fact: pytest (3.8.2) depends on more-itertools (>=4.0.0)
   1: fact: pytest (3.8.2) depends on attrs (>=17.4.0)
   1: fact: pytest (3.8.2) depends on six (>=1.10.0)
   1: fact: pytest (3.8.2) depends on py (>=1.5.0)
   1: selecting pytest (3.8.2)
   1: derived: py (>=1.5.0)
   1: derived: six (>=1.10.0)
   1: derived: attrs (>=17.4.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: atomicwrites (>=1.0)
   1: derived: pluggy (>=0.7)
   1: derived: funcsigs (*)
   1: derived: pathlib2 (>=2.2.0)
   1: derived: colorama (*)
PyPI: No release information found for py-0.8.0-alpha2, skipping
PyPI: No release information found for py-0.9.0, skipping
PyPI: No release information found for py-1.4.32.dev1, skipping
PyPI: 5 packages found for py >=1.5.0
PyPI: 2 packages found for six >=1.10.0
PyPI: 3 packages found for attrs >=17.4.0
PyPI: 5 packages found for more-itertools >=4.0.0
PyPI: 5 packages found for atomicwrites >=1.0
PyPI: 1 packages found for pluggy >=0.7
PyPI: 7 packages found for funcsigs *
PyPI: 4 packages found for pathlib2 >=2.2.0
PyPI: 37 packages found for colorama *
PyPI: Getting info for pluggy (0.7.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: pluggy-0.7.1-py2.py3-none-any.whl
   1: selecting pluggy (0.7.1)
PyPI: Getting info for six (1.11.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: six-1.11.0-py2.py3-none-any.whl
   1: selecting six (1.11.0)
PyPI: Getting info for attrs (18.2.0) from PyPI
   1: selecting attrs (18.2.0)
PyPI: Getting info for pathlib2 (2.3.2) from PyPI
   1: fact: pathlib2 (2.3.2) depends on scandir (*)
   1: fact: pathlib2 (2.3.2) depends on six (*)
   1: selecting pathlib2 (2.3.2)
   1: derived: scandir (*)
PyPI: 15 packages found for scandir *
PyPI: Getting info for py (1.6.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: py-1.6.0-py2.py3-none-any.whl
   1: selecting py (1.6.0)
PyPI: Getting info for more-itertools (4.3.0) from PyPI
   1: fact: more-itertools (4.3.0) depends on six (>=1.0.0,<2.0.0)
   1: selecting more-itertools (4.3.0)
PyPI: Getting info for atomicwrites (1.2.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: atomicwrites-1.2.1-py2.py3-none-any.whl
   1: selecting atomicwrites (1.2.1)
PyPI: Getting info for funcsigs (1.0.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: funcsigs-1.0.2-py2.py3-none-any.whl
   1: selecting funcsigs (1.0.2)
PyPI: Getting info for scandir (1.9.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: scandir-1.9.0.tar.gz
PyPI: Downloading wheel: scandir-1.9.0-cp27-cp27m-win32.whl
   1: selecting scandir (1.9.0)
PyPI: Getting info for colorama (0.3.9) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: colorama-0.3.9-py2.py3-none-any.whl
   1: selecting colorama (0.3.9)
   1: Version solving took 8.689 seconds.
   1: Tried 1 solutions.


Package operations: 8 installs, 0 updates, 0 removals

Writing lock file

  - Installing six (1.11.0)
  - Installing atomicwrites (1.2.1)
  - Installing attrs (18.2.0)
  - Installing more-itertools (4.3.0)
  - Installing pluggy (0.7.1)
  - Installing py (1.6.0)
  - Installing petl (1.2.0)
  - Installing pytest (3.8.2)
poetry add -vvv petl  7,50s user 2,10s system 36% cpu 26,646 total

As you can see, it takes less than 30 seconds in total to resolve the dependencies and install them.

I am also on MacOS 10.14.

The only thing I can think of is that you have trouble contacting either PyPI or files.pythonhosted.org where the files are hosted.

[cprieto]

That is weird, a simple ping to files.pythonhosted.org or PyPI didn't help, but after doing a simple traceroute I found really weird jumps and delays so I think it is something really really wrong with my internet connection (or at least the way they route the packages).

Is there any way I can tell poetry to use a mirror in the same way pip does with --use-mirrors? maybe in that way I can diagnose what is happening and see if the situation improves.

[cprieto]

I actually think it was something wrong with my connection since yesterday, I just cleared the cache and run it again and I got much better numbers:

https://gist.github.com/cprieto/63274728f066403cfed7c0fd58b33548

It went down to less than 6 seconds.

I was thinking adding some flag or way to diagnose problems like this, it sounds weird to me but to be honest it is not the first time this happens (at least for me), maybe something in the debug flag to see the performance call or output the request/response between poetry and pypi/pythonhosted?

I don't know, I want to help to whoever has a problem like this in the future.

[Zac-HD]

This is not an issue with Poetry, it's a problem with your internet connection and IMO that should be handled by other tools (e.g. a speed test, traceroute, etc), not Poetry.

[cjw296]

@cprieto - any chance you could close this issue? Trying to help @sdispater with issue cleanup :-)

[cprieto]

Yes, time to close this

[floer32]

[for posterity]

Even aside from network or PyPI issues, sometimes clearing out the cache from filelock/cachecontrol, can help. (tangentially related to #832 perhaps)

[cmoutafidis]

Not sure why this issue is closed, clearly there is something wrong with poetry
$poetry install
Updating dependencies
Resolving dependencies... (64103.8s)

I left my computer open one day and this still did not work

[mahendranbhathp]

I am also facing same issue with poetry 1.2.0

[clintonroy]

Please people, don't comment on closed bugs,. If you have a problem, please fill out the bug report, and give us all the details that we actually need to debug a problem. Saying your setup is slow does not help at all.