Skip to content

Loading…

do not walk on whole package for egg-info dir #853

Closed
wants to merge 1 commit into from

2 participants

@cenkalti

If only the top level *.egg-info dir is going to be picked,
then there is no need to walk after the first one is found
because os.walk() runs in top-down direction.

I did this because, we are deploying our application via git then installing in editable mode with "pip install -e .". Since there are thousands of small files and directories in our package, install command takes too much time, especially when there is a heavy load on the server.

I run the tests and these 2 have passed after my patch:

cenk@aslan ~/projects/pip/tests (develop*) $ nosetests -s -v test_basic.py:test_install_from_local_directory test_basic.py:test_download_editable_to_custom_path
Test installing from a local directory. ... ok
Test downloading an editable using a relative custom src folder. ... ok

----------------------------------------------------------------------
Ran 2 tests in 4.799s

OK

I cannot run all of them because Travis-CI is failing currently. I think these 2 should cover my patch.

@cenkalti cenkalti do not walk on whole package for egg-info dir
If only the top level *.egg-info dir is going to be picked,
then there is no need to walk after the first one is found
because os.walk() runs in top-down direction.
98c0bca
@cenkalti

Any comments on this?

@msabramo

Hmm, never got any comments. Maybe it can be closed?

@cenkalti cenkalti closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 19, 2013
  1. @cenkalti

    do not walk on whole package for egg-info dir

    cenkalti committed
    If only the top level *.egg-info dir is going to be picked,
    then there is no need to walk after the first one is found
    because os.walk() runs in top-down direction.
Showing with 33 additions and 32 deletions.
  1. +33 −32 pip/req.py
View
65 pip/req.py
@@ -292,40 +292,41 @@ def egg_info_path(self, filename):
base = self.source_dir
else:
base = os.path.join(self.source_dir, 'pip-egg-info')
- filenames = os.listdir(base)
- if self.editable:
- filenames = []
- for root, dirs, files in os.walk(base):
- for dir in vcs.dirnames:
- if dir in dirs:
- dirs.remove(dir)
- # Iterate over a copy of ``dirs``, since mutating
- # a list while iterating over it can cause trouble.
- # (See https://github.com/pypa/pip/pull/462.)
- for dir in list(dirs):
- # Don't search in anything that looks like a virtualenv environment
- if (os.path.exists(os.path.join(root, dir, 'bin', 'python'))
+
+ # if we have more than one match, we pick the toplevel one.
+ # This can easily be the case if there is a dist folder which
+ # contains an extracted tarball for testing purposes.
+ found = False
+ for root, dirs, files in os.walk(base):
+ for dir in vcs.dirnames:
+ if dir in dirs:
+ dirs.remove(dir)
+
+ # Iterate over a copy of ``dirs``, since mutating
+ # a list while iterating over it can cause trouble.
+ # (See https://github.com/pypa/pip/pull/462.)
+ for dir in list(dirs):
+ # Don't search in anything that looks like a virtualenv environment
+ if (os.path.exists(os.path.join(root, dir, 'bin', 'python'))
or os.path.exists(os.path.join(root, dir, 'Scripts', 'Python.exe'))):
- dirs.remove(dir)
- # Also don't search through tests
- if dir == 'test' or dir == 'tests':
- dirs.remove(dir)
- filenames.extend([os.path.join(root, dir)
- for dir in dirs])
- filenames = [f for f in filenames if f.endswith('.egg-info')]
-
- if not filenames:
+ dirs.remove(dir)
+
+ # Also don't search through tests
+ if dir == 'test' or dir == 'tests':
+ dirs.remove(dir)
+
+ if dir.endswith('.egg-info'):
+ found = True
+ self._egg_info_path = os.path.join(base, root, dir)
+ break
+
+ if found:
+ break
+
+ if not found:
raise InstallationError('No files/directories in %s (from %s)' % (base, filename))
- assert filenames, "No files/directories in %s (from %s)" % (base, filename)
-
- # if we have more than one match, we pick the toplevel one. This can
- # easily be the case if there is a dist folder which contains an
- # extracted tarball for testing purposes.
- if len(filenames) > 1:
- filenames.sort(key=lambda x: x.count(os.path.sep) +
- (os.path.altsep and
- x.count(os.path.altsep) or 0))
- self._egg_info_path = os.path.join(base, filenames[0])
+ assert found, "No files/directories in %s (from %s)" % (base, filename)
+
return os.path.join(self._egg_info_path, filename)
def egg_info_lines(self, filename):
Something went wrong with that request. Please try again.