New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools.find_packages() is much slower than it needs to be. #412

Closed
bb-migration opened this Issue Jul 20, 2015 · 2 comments

Comments

Projects
None yet
1 participant
@bb-migration

bb-migration commented Jul 20, 2015

Originally reported by: spookylukey (Bitbucket: spookylukey, GitHub: spookylukey)


find_packages effectively does ls -Ra on the directory, and then filters the result. If you happen to have a lot of files in the directory (e.g. for a checkout directory of a Python package under development) this can take a really long time.

If you need convincing that this might be important, read on, otherwise just wait for my pull request :-)

In particular, it badly affects tox - a common Python testing tool used in development that runs tests against various versions of dependencies. tox creates virtual environments for each environment to test, and typically uses a local setup.py sdist (or similar) to install the package into the virtualenv, which can can run find_packages if the setup.py uses that.

These virtualenvs (of which there can easily be dozens) are stored in a .tox folder in the local package checkout folder, meaning that you end up with many thousands of files in the package directory, and these are iterated over (multiple times) when tox is run. From a cold startup, I'm often waiting minutes for tox to run, and even with a warmed file system it takes often 10 seconds.

tox is not the only reason why the local dir could contain lots of files - other things like git/hg mean there can be many files in the local directory.

Issue #326 might be to do with this, but the original reporter seemed to think otherwise.

Thankfully, the issue is easily fixed - these hidden dirs are filtered away by the code, and all we need to do is filter them away earlier so that the recursive walk short-circuits. Pull request to follow.


@bb-migration

This comment has been minimized.

bb-migration commented Jul 21, 2015

Original comment by spookylukey (Bitbucket: spookylukey, GitHub: spookylukey):


See pull request #140.

@bb-migration

This comment has been minimized.

bb-migration commented Aug 7, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Update changelog. Fixes #412.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment