Package locking is crazy slow for scikit-learn #1785
Comments
|
you shouldn't be running lock in docker in the first place… |
|
I don't see why not. But that aside there's still a performance issue here I'd like to figure out. |
|
I've added --skip-lock to my compose file for now. Let me know how I can help debug this. |
|
@kennethreitz Running language-specific tools in Docker is normal. Specifically:
The goal here is that devs never need to worry about local/system Python versions, pipenv versions, etc. I'd also love to help debug this. |
|
Cheers @jleclanche I used to play hearthstone in its early days before there was much tooling around it so thanks for building some community tools! Generally speaking when you run If you run Resolvers are hard, and sometimes in python we need to download the whole package to parse it’s |
:)
I fully understand what pipenv brings to the table. Just to explain why I'm using it in docker:
With that said, I'm not interested in solving my problem. I solved my problem by adding --skip-lock. I'm interested in solving, or helping solve, the egregious difference in performance between inside and outside of the container. Or at least coming out of this with a "there's a very good reason for this difference and here it is". But yarn is also running inside that same container and managing 1-2 orders of magnitude more dependencies than pipenv, so I think we can do better. And if that takes me PRing setup.py/setup.cfg fixes to 30 different projects so be it :) |
|
@jleclanche - just to get at it why the docker version is slower than the local version, can you try running the command twice within the container? Generally, my assumption is that the time is being taken up in downloading/caching all the wheels (and possibly also building wheels too). i.e., I'm relatively certain the second / third / fourth times will be much faster. On my system (outside of Docker), when I bust my wheel/pipenv/pip caches, locking goes from taking 1.5 minutes to taking 5-10 minutes. If it is much faster the second (or third!) time, then the issue becomes how to persist the cache locally to get to that faster speed. In my (totally unofficial) opinion, I think you should be able to do this by explicitly setting the Semi-crazy cache-maintaining solution for dockerIf the slowness is from (lack of) caching between runs, something like the below might work (you'll need to have a pre-existing Pipfile.lock in your working directory so you can dump that out at the end): |
|
Uh, what's going on? |
|
Huh, looks like Pipenv really doesn't like the Pipfile being in |
|
mmhm. |
|
@jleclanche - so I guess possible (again, semi-hacky) solution would be to generate your lockfile separately using docker run that mounts your local filesystem (thus allowing you to reuse your cache) and then adding |
|
That's essentially what I'm doing right now, by having the venv live in the project and not be regenerated between containers. But again, I'm good on my side, i just want to figure out where exactly the performance regression is happening :) |
|
If you attach to the container while it’s locking, can you see what it’s doing? How’s your *nix-fu? We can check processes, open files, network connections, load |
|
My nix-fu is alright, my docker-fu not so much. I'll try to debug this some more and see what I can find out. I don't think it's all network because it takes super long even with a local pypi mirror. |
|
If you can spin up the docker container you can have it just idly run bash or whatever. Then I think you can attach to it and run screen or some such and run your pipenv stuff in one window and watch stuff in the other |
|
@jleclanche - just for context: if you have a pre-existing Pipfile.lock, how long does it take to just install it directly? I.e. something like: I believe that's the minimal baseline for timing (because it's just time for |
|
Yeah, it's considerably faster like that: |
|
@jleclanche - that's with zero cache? (i.e., everything is already downloaded, etc) |
|
err, nothing is already downloaded, no pipenv or pip use except for |
|
Yes, fresh container + |
|
wow, that's super fast - guessing that's with local PyPI instance or something? But also really helpful to separate out where slowness is coming from :) |
|
Both of those commands are with regular pypi. I think running the command again might not reinstall though, so it's probably faster than an actual reinstall would be. |
|
Ok, good call on the cache. Pipfile: [[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
scikit-learn = {extras = ["alldeps"]}
[requires]
python_version = "3.6"Time in docker: Time on host after removing ~/.cache (different system/setup though): So I guess it wasn't a docker thing after all; scikit-learn was getting cached systemwide, which wasn't happening for docker. I'm reinvestigating the network stuff, because my pip cache is now 1.1G and I'm starting to think my local mirror setup wasn't working in docker. For the record, replacing |
jleclanche
changed the title
|
I'm leaving the debugging here, have to take care of RL for a while -- tough getting network logging set up here, I might be better off using wireshark... The good news is it's now apparent why re-locking in docker would take so long: Because ~/.cache is ephemeral in docker, every pipenv call that uses that cache would have to redownload everything. I still think there is a legitimate issue here though: If both the lockfile and the venv are present and set up, pipenv should not have to redownload anything, even if there's no cache. |
|
@jleclanche Ah, that makes sense. Thanks for the hard work! Unfortunately Pipenv really needs to download things because how Python’s packaging system currently works. There’s no way to know what dependencies a package has without actually running its setup.py, and the only way to get the file is from PyPI (or the cache). We would need to jump through a lot of hoops to prevent this from happening, if it is at all possible to prevent. Can you verify whether mounting the host’s cache directory into Docker works (using |
On first download, sure, but that first download is fine anyway because you have to download the package somehow. How about adding the list of each package's dependencies to Pipfile.lock? That seems to be the missing key here. Coming back to yarn, that is in fact exactly what yarn does: ...
aws4@^1.2.1, aws4@^1.6.0:
version "1.6.0"
resolved "https://registry.yarnpkg.com/aws4/-/aws4-1.6.0.tgz#83ef5ca860b2b32e4a0deedee8c771b9db57471e"
babel-code-frame@^6.22.0, babel-code-frame@^6.26.0:
version "6.26.0"
resolved "https://registry.yarnpkg.com/babel-code-frame/-/babel-code-frame-6.26.0.tgz#63fd43f7dc1e3bb7ce35947db8fe369a3f58c74b"
dependencies:
chalk "^1.1.3"
esutils "^2.0.2"
js-tokens "^3.0.2"
babel-core@^6.26.0:
version "6.26.0"
resolved "https://registry.yarnpkg.com/babel-core/-/babel-core-6.26.0.tgz#af32f78b31a6fcef119c87b0fd8d9753f03a0bb8"
dependencies:
babel-code-frame "^6.26.0"
babel-generator "^6.26.0"
babel-helpers "^6.24.1"
babel-messages "^6.23.0"
babel-register "^6.26.0"
babel-runtime "^6.26.0"
babel-template "^6.26.0"
babel-traverse "^6.26.0"
babel-types "^6.26.0"
babylon "^6.18.0"
convert-source-map "^1.5.0"
debug "^2.6.8"
json5 "^0.5.1"
lodash "^4.17.4"
minimatch "^3.0.4"
path-is-absolute "^1.0.1"
private "^0.1.7"
slash "^1.0.0"
source-map "^0.5.6"
babel-generator@^6.26.0:
version "6.26.1"
resolved "https://registry.yarnpkg.com/babel-generator/-/babel-generator-6.26.1.tgz#1844408d3b8f0d35a404ea7ac180f087a601bd90"
dependencies:
babel-messages "^6.23.0"
babel-runtime "^6.26.0"
babel-types "^6.26.0"
detect-indent "^4.0.0"
jsesc "^1.3.0"
lodash "^4.17.4"
source-map "^0.5.7"
trim-right "^1.0.1"
babel-helper-builder-binary-assignment-operator-visitor@^6.24.1:
version "6.24.1"
resolved "https://registry.yarnpkg.com/babel-helper-builder-binary-assignment-operator-visitor/-/babel-helper-builder-binary-assignment-operator-visitor-6.24.1.tgz#cce4517ada356f4220bcae8a02c2b346f9a56664"
dependencies:
babel-helper-explode-assignable-expression "^6.24.1"
babel-runtime "^6.22.0"
babel-types "^6.24.1"
... |
Bind mounts. It would (probably) work, but actually there's other issues with that. Namely, the docker service runs as root (meaning it writes files as root), and the container itself runs root as well (as is common). Pip will refuse to use a cache from a mismatching UID, so mounting ~1000/.cache/pip will do nothing unless the container itself has a UID 1000, and even if it does, it will write files as root which is not something you want. |
|
I usually use pipenv in docker and indeed, you do not lock inside the docker. You run --deploy to build your application from the pipfile.lock. If you want to update the lock, you run pipenv update/lock from outside of the docker (what is the use of running inside the docker?), but you do it left often and in a controlled way (you really need to check the locked file afterwards) and run a new set of non regression tests. Once commited, then you run your app with the updated lock file. TL;DR: lock the dependencies of your app from your dev env, run from within a docker using PS: this should maybe be better document in the pipenv docs, lot of people make this mistake, no? PPS: I tend to favour |
|
@gsemet Docker is my dev env. I'm not using pipenv in production at the moment (and once I am, I will be following that workflow indeed). I agree with the premise and with documenting it but let's keep this issue on topic. I and the whole python community alike all want pipenv to be blazing fast if we're going to use it daily :) Waiting to hear some thoughts re. adding dependencies to the lockfile. |
|
I +1'ed @jleclanche's most recent comment, but to be specific on which part: Docker as a dev env for Python is a very powerful idea. Pipenv is like the missing link that makes this workflow plausible, so going forward, it would be great to understand the cause of the apparent perf issue. |
|
I first saw that in the case of Traefik, to carry all the go deps and build tools in a repetitive way. I'd like to better understand what the "docker as dev env" approach has that a traditional pipenv/virtualenv does not provide already. Except for system dependencies that cannot be packaged inside the venv (ex: python gtk/dbus/...), when correctly controlled, pipenv and virtualenv can provide a fully reproducible environment. |
I'll invite you to look at the environment I posted in the original issue. It is about system dependencies. Happy to discuss docker further by email if you have questions but at this point I'd like to ask people to keep it out of this particular Github thread and stay on topic. |
|
@jleclanche the thing is, explicitly calling Now what I said in the paragraph above was that explicit calls to The concern I would have here is if we ever decide to flatten the dependency graph to sub-dependencies, we're right back to square one. |
|
Something we would like to be faster is I don't think that implies storing hierarchical dependency metadata, though - we just need a cache of "project-name-and-version" -> "declared-dependencies" metadata, similar to the way the lock file already stores "project-name-and-version" -> "expected-artifact-hashes". That way when |
|
@ncoghlan under that approach, what happens if we install a new package and run into a dependency conflict? Should we be caching the per-top-level-package graph somewhere as well with the un-pinned specs so that we can check to see if conflicts can be resolved without re-downloading? |
|
@techalchemy I'd personally separate more efficient conflict resolution out as a "later" problem, since you're getting into the same kind of territory as https://github.com/pradyunsg/zazo at that point. The initial naive cache wouldn't help with that, it would purely be about making "Get me the declared dependencies for version X of project Y" faster when both the previous lock file and the updated lock file include the same version. In the |
|
Even if not inside docker locking packages is too slow. this was fast af during pipenv v9. not only in scikit even installing other packages are affected too. |
|
@exequiel09 I’m not sure this was ever exactly fast. The tradeoff we made was to make sure we include dependencies from I am not sure how we can speed this up but it is quite slow I agree |
|
@jleclanche FYI if you copy your pipenv cache ( |
|
Closing for now -- recommendation is to mount persistent cache volumes for pip and pipenv |
|
For the record, copying the cache isn't enough because of permissions. What I've done is share the project folder, then put the .venv in the project with PIPENV_VENV_IN_PROJECT and set XDG_CACHE_HOME to that venv: environment:
PIPENV_VENV_IN_PROJECT: "1"
XDG_CACHE_HOME: "/path/to/project/source/.venv/.cache" |
|
Note that |
|
@jleclanche There is |
I'm sorry I don't have a reduced test case for this but this is so crazy slow it's hard to actually debug.
Steps to reproduce
docker-compose build django(this builds an image based on the python-stretch docker image, which will also install the latest pipenv systemwide, cf. Dockerfile).docker-compose run django, which runspipenv install --devOn linux, this stays stuck at
Locking [packages]for over 15 minutes, with no output even when run with--verbose. Then after ~15 mins, it gives me the full output of what it's been doing for all that time.When run outside of docker, it still takes a couple of minutes on that step, but at most 1-2 mins. I have a pretty beefy CPU and SSD, so I don't know why it would take this long in the first place.
I also see a lot of
Warning: Error generating hash for ...in the verbose output, I don't know if that's related.Any idea? How can I debug this further?
The text was updated successfully, but these errors were encountered: