Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Lock updating is very slow #1914
Updating a lockfile can be very slow. A standard
This means every time I need to install, uninstall or upgrade a package I need to take a 2-minute break to wait for pipenv to finish updating its lockfile. I'm not sure why that is; yarn and npm seem to perform a similar task but only take seconds, even for projects that have many many more dependencies.
npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass
Closing to track in the other issues.
No. Pipenv is an application depedency resolution tool. When a dependency is used as a Python package by another application, it is no longer an application. Its dependencies are determined by setup.py (or setup.cfg or whatever you use to upload to PyPI). Loading dependencies from a lock file is a sure route to dependency hell, we will likely never ever do that.
As far as I gather, the overwhelming benefit of pipenv is the assurance dependencies will play nicely together — not just for you, but for anyone later dealing in your code. The product of that assurance, the lock file, absolutely takes more time than anyone expects or desires, including the devs — see #2200.
However, I think you can also understand the opportunity pipenv has to shepherd well-meaning devs in the Python community at large toward a workflow imposing less head-scratching on future contributors — who might've only been visitors had they given up during the "figure out how to setup the dev environment" stage; and less hands thrown up by future maintainers — who might've only been drive-by PR authors had they given up during the "seriously screwing around with deep project internals" stage.
Better to leave it available only as an env var, or some other method whose application rests squarely in the "your user-specific local config, your fault" territory, allowing pipenv to overcome the passing phase of lockfile generation slowness without giving up the truly beneficial cultural shift toward explicitness over implicitness in package management.
Python's incredibly vast standard library is an enormous asset, whose history has undergone many eras of imposing consistency. That most standard packages play nicely together is an enormous feat involving consideration over many years by many people. One day, that play-nice-ability will extend to most Python projects encountered on the web — far, far from the stdlib, and with far, far fewer PEPs required (and far, far fewer BDFLs vacating in frustration). The impact of such a unilaterally buttery experience is hard to measure, but some current languages did refuse to compromise conceptual integrity for immediate convenience... and oh, the places they'll Go.
So yes, generating the lockfile is slow, and yes, it's frustrating when you only wanted
Lockfile generation is slow only because it's making explicit what we've all taken for granted. But because it hurts, we will adjust things so it doesn't. We broke our arm because we pushed ourselves doing things we believed in. Sure, we can avoid the pain by never using that arm again— or we can put it in a cast while it heals.
I'd be the last person to tell you not to make pipenv convenient for yourself today (otherwise I'd be a shitty developer — though, the jury's still out), but I implore you to see the frustration of lockfile generation as a growing pain while the Python community develops into a strong, healthy body with more fully-functioning limbs than one really expected when removing a cast. We made it to Python 3. Let's make it to dependency management in the stdlib.
Pandas-datareader install fails on first attempt, possibly cause of
Using version 2018.11.26
$ pipenv --version pipenv, version 2018.11.26
Results: 144.82 real 33.68 user 5.77 sys
$ time pipenv install -r requirements.txt Requirements file provided! Importing into Pipfile… Pipfile.lock (0fdb67) out of date, updating to (a65489)… Locking [dev-packages] dependencies… Locking [packages] dependencies… ✔ Success! Updated Pipfile.lock (0fdb67)! Installing dependencies from Pipfile.lock (0fdb67)… 🎅 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 15/15 — 00:00:04 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. 144.82 real 33.68 user 5.77 sys
Results: 4.54 real 5.33 user 0.87 sys
$ time pipenv install -r requirements.txt --skip-lock Requirements file provided! Importing into Pipfile… Installing dependencies from Pipfile… 🎅 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:01 To activate this project's virtualenv, run pipenv shell. Alternatively, run a command inside the virtualenv with pipenv run. 4.54 real 5.33 user 0.87 sys
It won't even finish on my machine:
@black-snow I'd recommend trying it in a different shell. Without diving too deep into things, it seems like pexpect (a library for programmatically interfacing with interactive CLI programs) is used to detect the shell pipenv is being executed from, and that might be stalling. pexpect kinda seems overkill for such a thing, but I'm not aware of the whole context.