Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poetry package install time seems longer than installing with pip #338

Open
MichaelAquilina opened this Issue Jul 26, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@MichaelAquilina
Copy link

MichaelAquilina commented Jul 26, 2018

  • I am on the latest Poetry version.
  • I have searched the issues of this repo and believe that this is not a duplicate.
    If an exception occurs when executing a command, I executed it again in debug mode (-vvv option). (Does not apply)

Issue

I am reasonably sure that installing packages using poetry install instead of pip install takes significantly longer.

To compare. Here is a pip install taking 35 seconds: https://circleci.com/gh/MichaelAquilina/S4/464#action-103

Here is another build with poetry install and the same requirements taking 1:27 seconds: https://circleci.com/gh/MichaelAquilina/S4/521#action-104

In both cases, both dev and non dev requirements were installed.

@sdispater

This comment has been minimized.

Copy link
Owner

sdispater commented Jul 26, 2018

Thanks for your interest in Poetry!

This is expected because Poetry orders the packages to install so that the deepest packages in the dependency graph are installed first to avoid errors at installation time. This requires sequential installation of the packages which takes longer but is more "secure".

Also, Poetry checks hashes of installed packages for security reasons and due to the way pip works, Poetry has to generate a temporary requirements.txt file to make pip check hashes. This adds an overhead which explains the difference between the two tools.

@MichaelAquilina

This comment has been minimized.

Copy link
Author

MichaelAquilina commented Jul 26, 2018

Thanks for taking the time to explain :)

Would it be possible for poetry to just download the target packages in parallel and install in sequence to speed up the process?

@sdispater

This comment has been minimized.

Copy link
Owner

sdispater commented Jul 26, 2018

I intend to improve the installation part of Poetry sometime in the future to speed things up, yes. I can't give you an ETA though, the only thing I can tell you is that it will likely be after the 1.0 milestone since I want to stabilize Poetry before changing critical parts, like this one.

I'll be sure to keep you posted if anything changes on that front.

@MichaelAquilina

This comment has been minimized.

Copy link
Author

MichaelAquilina commented Jul 26, 2018

That sounds great @sdispater. Of course I fully understand your reasoning and agree that ensuring stability is a lot more important than performance.

Thanks for the great work on Poetry!

@dbarrosop

This comment has been minimized.

Copy link

dbarrosop commented Dec 27, 2018

Let me start by saying I recently discovered poetry and I think it's awesome :)

Now, is there any progress on this? I just migrated a couple of projects to poetry and my container builds went from 2s to 5 minutes each. The reason why it's so slow is twofold:

  1. When building containers I'd do something like this:
ADD requirements.txt /tmp
RUN pip install -r requirements.txt

ADD . /project
...

so I could cache that layer. With poetry I have to do:

ADD ./project
RUN cd /project && poetry install

Which means I need to install the dependencies over and over even though I haven't changed a single dependency. I am trying to do something equivalent with:

ADD poetry.lock /tmp
ADD pyproject.toml /tmp
RUN cd /tmp && /root/.poetry/bin/poetry update --no-dev

But I am not sure if this is the right thing to do but does the job. Feels extremely hacky so I'd appreciate some advice :P

  1. The second issue is the one described here. A clean install takes ~30s with pip and ~5minutes with poetry :( Monitoring the network I can see how pip downloads everything very fast at a few MB/s while poetry does it one by one never reaching more than a dozen KB/s.
@DanCardin

This comment has been minimized.

Copy link

DanCardin commented Jan 28, 2019

Re this previous comment, it would be ideal if there was an interface to install when only the lock file is available, so that we can continue to make good use of docker layer caching. Because I haven't been able to get an install to succeed when the source isn't also available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.