Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Have a way to ingore specific dirs when building list of files in egg_info #249
I have a very large
I had a similar problem and finally tracked it down: symlinks inside my project to a different filesystem when running inside a local vmware machine. (Background: buildout's "omelette" recipe that provides symlinks inside a project's parts/omelette folder).
Slowness all because setuptools wants to create a full list of all files. Even though, in my case, there is a MANIFEST.in that tells it to grab the local *.rst files and everything in one specific subdirectory.
Note that here's a monkeypatch you can apparently apply locally: https://mail.python.org/pipermail/distutils-sig/2015-April/026175.html
It is starting to look attractive :-) But I don't feel like that's the best way.
Would it be possible to automatically exclude at least directories in setup.py's directory when they're excluded in the MANIFEST.in? It would mean some of the exclusion-code would run before and after walking the directory, but that's preferable (to me) to waiting a couple of minutes or even an hour.
Yes, I've had projects where "setup.py develop ." took an hour! One of the subdirectories was an remote CIFS mount on storage that's used for archive storage.
I've got a monkeypatch (https://gist.github.com/reinout/b453e6fa98289b1a1983) which you can import from the top of your setup.py. This fixes the slowness.
referenced this issue
Mar 29, 2016
I'm thinking that a totally new approach to handling the MANIFEST.in file is the answer.
The current approach (inherited from distutils) is to recursively list all the files, bringing the whole lot into memory, and then apply MANIFEST.in commands to that list. This is the problem, and the fix is "don't do that".
It is just the wrong approach. It gives you a small speedup in one case - when you have very few files, and you end up processing MANIFEST.in directives that would cause multiple walks over the directory structure. However, in this case, OS caching means that you the second walk is usually going to be very fast.
A far better approach is to just handle MANIFEST.in directives directly - for example, a
There are two ways to go about this:
My suggestion is that this should be implemented within setuptools. If successful, it could be submitted to distutils etc. as a replacement implementation.