Have a way to ingore specific dirs when building list of files in egg_info #249
Comments
Original comment by reinout (Bitbucket: reinout, GitHub: reinout): I had a similar problem and finally tracked it down: symlinks inside my project to a different filesystem when running inside a local vmware machine. (Background: buildout's "omelette" recipe that provides symlinks inside a project's parts/omelette folder). But I've seen the same slowness when there's a filesystem cache directory inside the project. And a similar one with a huge number of javascript dependencies that are downloaded locally into the project folder (via 'bower' and 'npm'). Slowness all because setuptools wants to create a full list of all files. Even though, in my case, there is a MANIFEST.in that tells it to grab the local *.rst files and everything in one specific subdirectory. |
Original comment by reinout (Bitbucket: reinout, GitHub: reinout): Note that here's a monkeypatch you can apparently apply locally: https://mail.python.org/pipermail/distutils-sig/2015-April/026175.html It is starting to look attractive :-) But I don't feel like that's the best way. Would it be possible to automatically exclude at least directories in setup.py's directory when they're excluded in the MANIFEST.in? It would mean some of the exclusion-code would run before and after walking the directory, but that's preferable (to me) to waiting a couple of minutes or even an hour. Yes, I've had projects where "setup.py develop ." took an hour! One of the subdirectories was an remote CIFS mount on storage that's used for archive storage. |
Original comment by reinout (Bitbucket: reinout, GitHub: reinout): I've got a monkeypatch (https://gist.github.com/reinout/b453e6fa98289b1a1983) which you can import from the top of your setup.py. This fixes the slowness. See http://reinout.vanrees.org/weblog/2016/03/17/setuptools-speed.html |
I'm thinking that a totally new approach to handling the MANIFEST.in file is the answer. The current approach (inherited from distutils) is to recursively list all the files, bringing the whole lot into memory, and then apply MANIFEST.in commands to that list. This is the problem, and the fix is "don't do that". It is just the wrong approach. It gives you a small speedup in one case - when you have very few files, and you end up processing MANIFEST.in directives that would cause multiple walks over the directory structure. However, in this case, OS caching means that you the second walk is usually going to be very fast. A far better approach is to just handle MANIFEST.in directives directly - for example, a There are two ways to go about this:
My suggestion is that this should be implemented within setuptools. If successful, it could be submitted to distutils etc. as a replacement implementation. |
Originally reported by: ionelmc (Bitbucket: ionelmc, GitHub: ionelmc)
I have a very large
.tox
dir, i'd like to have a way so that setuptools ignores it completely. Now it recurses inside it and takes few minutes to run egg_info and other commands that rely on distutils.filelist.listallThe text was updated successfully, but these errors were encountered: