New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock updating is very slow #1914

Closed
wichert opened this Issue Apr 5, 2018 · 24 comments

Comments

Projects
None yet
@wichert
Copy link

wichert commented Apr 5, 2018

Updating a lockfile can be very slow. A standard pipenv lock easily takes well over a minute for me:

$ time pipenv lock 
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (abef76)!

real	1m56.988s
user	0m21.805s
sys	0m2.417s

This means every time I need to install, uninstall or upgrade a package I need to take a 2-minute break to wait for pipenv to finish updating its lockfile. I'm not sure why that is; yarn and npm seem to perform a similar task but only take seconds, even for projects that have many many more dependencies.

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented Apr 5, 2018

We are aware and have many issues tracking this topic. See #1785 #1886 #1891 and PR #1896

npm and yarn have the advantage of not having to fully download and execute each prospective package to determine their dependency graph because the dependencies are specified in plaintext. Python dependencies require us to fully download and execute the setup files of each package to resolve and compute. That's just the reality, it's a bit slow. If you can't wait 2 minutes or you feel it's not worth the tradeoff, you can always pass --skip-lock.

Closing to track in the other issues.

@techalchemy techalchemy closed this Apr 5, 2018

@djangorobert

This comment has been minimized.

Copy link

djangorobert commented Jul 4, 2018

Same here im trying it out right now and installing django is already at 10 minutes and going

@djangorobert

This comment has been minimized.

Copy link

djangorobert commented Jul 4, 2018

seems that if you try and install a couple at a time its slow but if you do one at a time works a bit faster

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented Jul 4, 2018

If you can provide a Pipfile that reproduces the slow behavior it would be helpful -- thanks

@dideler

This comment has been minimized.

Copy link

dideler commented Jul 17, 2018

Python dependencies require us to fully download and execute the setup files of each package to resolve and compute

Is this still the case when the package has its own Pipfile.lock, or will pipenv use that when possible to determine dependencies?

@uranusjr

This comment has been minimized.

Copy link
Member

uranusjr commented Jul 17, 2018

will pipenv use that when possible to determine dependencies?

No. Pipenv is an application depedency resolution tool. When a dependency is used as a Python package by another application, it is no longer an application. Its dependencies are determined by setup.py (or setup.cfg or whatever you use to upload to PyPI). Loading dependencies from a lock file is a sure route to dependency hell, we will likely never ever do that.

@iddan

This comment has been minimized.

Copy link

iddan commented Aug 9, 2018

It's still super slow

@uranusjr

This comment has been minimized.

Copy link
Member

uranusjr commented Aug 9, 2018

@iddan Thanks for the reminder, Captain Obvious!

@iddan

This comment has been minimized.

Copy link

iddan commented Aug 9, 2018

Sorry, as an OSS maintainer I know how sometimes issues can get dismissed because of age. Just wanted to state that even the issue received discussion it’s still relevant

@statueofmike

This comment has been minimized.

Copy link

statueofmike commented Aug 13, 2018

@techalchemy note of --skip-lock above is wonderful. This should be a more accessible or publicized option. Can we set it as a default somewhere?

@CauchyPolymer

This comment has been minimized.

Copy link

CauchyPolymer commented Oct 1, 2018

@techalchemy what if it takes 20 minutes with my brand new mac pro?

@theY4Kman

This comment has been minimized.

Copy link

theY4Kman commented Oct 13, 2018

@techalchemy note of --skip-lock above is wonderful. This should be a more accessible or publicized option. Can we set it as a default somewhere?

As far as I gather, the overwhelming benefit of pipenv is the assurance dependencies will play nicely together — not just for you, but for anyone later dealing in your code. The product of that assurance, the lock file, absolutely takes more time than anyone expects or desires, including the devs — see #2200.

However, I think you can also understand the opportunity pipenv has to shepherd well-meaning devs in the Python community at large toward a workflow imposing less head-scratching on future contributors — who might've only been visitors had they given up during the "figure out how to setup the dev environment" stage; and less hands thrown up by future maintainers — who might've only been drive-by PR authors had they given up during the "seriously screwing around with deep project internals" stage.

If --skip-lock were to become a permanent flag in a Pipfile or a setting in a pipenv config, pipenv's perception would slowly slide toward "better pip", and just another stepping stone fading into the horizon as the community eventually landed on a less compromising spiritual successor.

Better to leave it available only as an env var, or some other method whose application rests squarely in the "your user-specific local config, your fault" territory, allowing pipenv to overcome the passing phase of lockfile generation slowness without giving up the truly beneficial cultural shift toward explicitness over implicitness in package management.

Python's incredibly vast standard library is an enormous asset, whose history has undergone many eras of imposing consistency. That most standard packages play nicely together is an enormous feat involving consideration over many years by many people. One day, that play-nice-ability will extend to most Python projects encountered on the web — far, far from the stdlib, and with far, far fewer PEPs required (and far, far fewer BDFLs vacating in frustration). The impact of such a unilaterally buttery experience is hard to measure, but some current languages did refuse to compromise conceptual integrity for immediate convenience... and oh, the places they'll Go.

So yes, generating the lockfile is slow, and yes, it's frustrating when you only wanted pip install --save. But it's only because we've been sweeping an elephant into the closet for years — believing we didn't have a tangled mess of expectations and intentions from external dependencies, because "it installed fine on my machine".

Lockfile generation is slow only because it's making explicit what we've all taken for granted. But because it hurts, we will adjust things so it doesn't. We broke our arm because we pushed ourselves doing things we believed in. Sure, we can avoid the pain by never using that arm again— or we can put it in a cast while it heals.

I'd be the last person to tell you not to make pipenv convenient for yourself today (otherwise I'd be a shitty developer — though, the jury's still out), but I implore you to see the frustration of lockfile generation as a growing pain while the Python community develops into a strong, healthy body with more fully-functioning limbs than one really expected when removing a cast. We made it to Python 3. Let's make it to dependency management in the stdlib.

@sjjpo2002

This comment has been minimized.

Copy link

sjjpo2002 commented Nov 5, 2018

This took 38 minutes on my machine to create the lock file. Running Windows 10 and using Python 3.7

@vanyakosmos

This comment has been minimized.

Copy link

vanyakosmos commented Nov 11, 2018

Only numpy and pillow were already installed and it took <1 seconds to install numba and 25 minutes to lock it. Does pipenv forcibly compile every lib on lock or how does this work?

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented Nov 12, 2018

FYI persistent skip lock is just waiting on someone to flip the switch for auto_envvar_prefix which is a click setting. I've been 100% focused on core functionality so I haven't had a chance to look at this yet but I suspect it isn't that difficult

@zazazack

This comment has been minimized.

Copy link

zazazack commented Dec 25, 2018

TLDR; Typical pipenv install invocation: Time: 144.82 real 33.68 user 5.77 sys. With --skip-lock: Time: 4.54 real 5.33 user 0.87 sys.

Pandas-datareader install fails on first attempt, possibly cause of lock hanging. Is this an issue anyone else is seeing with other packages?

Using version 2018.11.26

$ pipenv --version
pipenv, version 2018.11.26

Contents of requirements.txt

sklearn-pandas
sklearn
pandas
numpy
matplotlib
statsmodels

Typical pipenv install invocation. Timed execution using time (BSD).

Results: 144.82 real 33.68 user 5.77 sys

$ time pipenv install -r requirements.txt
Requirements file provided! Importing into Pipfile…
Pipfile.lock (0fdb67) out of date, updating to (a65489)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (0fdb67)!
Installing dependencies from Pipfile.lock (0fdb67)…
  🎅   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 15/15 — 00:00:04
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
      144.82 real        33.68 user         5.77 sys

Invoking w\ --skip-lock

Results: 4.54 real 5.33 user 0.87 sys

$ time pipenv install -r requirements.txt --skip-lock
Requirements file provided! Importing into Pipfile…
Installing dependencies from Pipfile…
  🎅   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
        4.54 real         5.33 user         0.87 sys
@jufemaiz

This comment has been minimized.

Copy link

jufemaiz commented Jan 8, 2019

I think https://github.com/pandas-dev/pandas/ may be a problem? It's a common point with the time for me too.

Although pytest may also be an issue :\

@black-snow

This comment has been minimized.

Copy link

black-snow commented Jan 8, 2019

It won't even finish on my machine:

Installing pandas…
Adding pandas to Pipfile's [packages]…
Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
  File "c:\python36\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 109, in expect_loop
    return self.timeout()
  File "c:\python36\lib\site-packages\pipenv\vendor\pexpect\expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x00000292ADCCCDD8>
searcher: searcher_re:
    0: re.compile('\n')
@theY4Kman

This comment has been minimized.

Copy link

theY4Kman commented Jan 11, 2019

@black-snow I'd recommend trying it in a different shell. Without diving too deep into things, it seems like pexpect (a library for programmatically interfacing with interactive CLI programs) is used to detect the shell pipenv is being executed from, and that might be stalling. pexpect kinda seems overkill for such a thing, but I'm not aware of the whole context.

@black-snow

This comment has been minimized.

Copy link

black-snow commented Jan 12, 2019

@theY4Kman thanks for the advice. pipenv is working fine on another pc with same ubuntu and bash version ...

@kstohr

This comment has been minimized.

Copy link

kstohr commented Jan 16, 2019

Read this while waiting for pipfile to lock...:) Would be great if there was a solution.

@uetchy

This comment has been minimized.

Copy link

uetchy commented Jan 17, 2019

Seems like we need some kind of API to parse and cache Python packages deps and distribute it in a machine-friendly format. So we will no longer need to download entire packages and parse them.

@jufemaiz

This comment has been minimized.

Copy link

jufemaiz commented Jan 17, 2019

I believe bundler and ruby gems maintains and uses somthing like that.

@danielcompton

This comment has been minimized.

Copy link

danielcompton commented Jan 17, 2019

Java also has POM files (XML) which contain package dependencies and other information about the package. They are uploaded separately from the compiled JARs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment