Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipenv very slow. Takes an hour to install and lock. #2873

Open
ScarletMcLearn opened this Issue Sep 20, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@ScarletMcLearn
Copy link

ScarletMcLearn commented Sep 20, 2018

Pipenv is very slow.

It takes a long time to download packages.

Install packages.

And lock file.

@Jamim

This comment has been minimized.

Copy link
Contributor

Jamim commented Sep 21, 2018

Unfortunately, this is true.

By the way, do you use some additional indexes (sources) in your Pipfile?
If so, that is the possible reason of the issue.
#2730 (comment)

@devkral

This comment has been minimized.

Copy link

devkral commented Oct 20, 2018

The "Poetry" (package manager) project has much faster hashing.
It would be cool if you would use their algorithm (or faster).

@devkral

This comment has been minimized.

Copy link

devkral commented Oct 20, 2018

And I don't get why something simple like --help needs 5 seconds.
Maybe some code pathes should be lazy loaded.

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented Oct 21, 2018

We don’t use a special algorithm for hashing, we use sha256. I suspect poetry just trusts the hashes it’s told if it’s significantly faster at hashing. We re-hash every file for security.

If it takes 5 seconds to run simple arguments that’s an issue and you should open an issue about it with details. We’d be glad to merge anything that gets us performance gains without costs to functionality

@devkral

This comment has been minimized.

Copy link

devkral commented Oct 22, 2018

Trusting the hashes? Sounds like a good idea if the hashes are checked when using the lock-file.
Pip checks the hashes too, doesn't it?
Anyway: do you request the packages sequentially? I suspect some loops, serial lookup and extensive object generation.
I think the slow help call is also a symptom of this.
When I scanned skimmed the code I saw that the vendor package is not a namespace package. Every request to vendor loads all packages in it => namespaces are lazy.

@devkral

This comment has been minimized.

Copy link

devkral commented Oct 22, 2018

Even worse: patched standard packages. And patched is also no namespace package.
def _patch_path(): loads all vendored packages. Again performance!
Use a lazy loader which patches and caches on the fly.

@devkral

This comment has been minimized.

Copy link

devkral commented Oct 22, 2018

@rooterkyberian

This comment has been minimized.

Copy link

rooterkyberian commented Oct 25, 2018

@devkral
using time: pipenv --help 0,45s user 0,07s system 99% cpu 0,522 total , so it is noticeable, but seeing how "rare" its to invoke pipenv without some even more time-consuming action (like installing packages) IMO its not a really a problem. Lazy loading would be nice, of course, but more benefit could be found in other places.

What I personally oppose is the notion to remove eye candies, that actually speed up lookup of information, just to get some milliseconds back of CPU time at the cost of milliseconds to seconds of human time (be it during use, or development/support time).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.